Abstract: Claims of large lifts in A/B tests are widespread, yet many are only supported by small online experiments that are likely underpowered. Trustworth...
When something goes horribly wrong mid-experiment, should you stick to the plan or stop the test? A story about two statisticians, and possibly a bear.
Practitioners learn the definitions of type-I and type-II errors and then promptly confuse them. Here is a story that might help.
Abstract: Underpowered studies (below 50% power) suffer from the winner’s curse: A statistically significant positive estimate must exaggerate the true treat...
Abstract: Organizations increasingly rely on experimentation to drive data-informed decision-making and innovation. While models like the Flywheel offer guid...
I was interviewed for the nohacks.show podcast (again) together with Erin Weigel.
I was interviewed for the From A to B podcast. We talked about tacos, as well as some other less important things.
I was interviewed for the growth-minded SUPERHEROES podcast.
I was interviewed for the Experiment Nation podcast.
I was interviewed for the nohacks.show podcast.
In the first two posts in this series, we explained what interaction effects are and what their consequences could be if they remain undetected. We also list...
In the previous post in this series, we defined interaction effects and discussed the potential consequences if they remain undetected. In this post, we will...
Experiments allow us to test how changes to our products affect the behavior of our users. We make many such changes in many experiments running at the same ...
I was interviewed for the Analytics Power Hour podcast.
This week I learnt that I have been confusing people for years about what a hypothesis really is. A conversation with Erin and Craig helped me understand my ...
I was interviewed for the Experimentation Masters podcast.
Speed is not velocity, but this does not mean that speed is not important. In fact, speed is a prerequisite for velocity: without speed, there can be no velo...
There is a popular meme which is used as a response to “useless” meetings: “this meeting could have been an email”. I think replacing meetings with emails wo...
“Won’t running all those experiments slow down our product development?” When speaking at conferences, I get this question a lot. And my answer is always the...
We are building a culture of experimentation at Vista. We are taking a scalable, decentralized approach, requiring people and teams across the organization t...
Abstract: A/B tests, or online controlled experiments, are heavily used in industry to evaluate implementations of ideas. While the statistics behind control...
SRM checks are straightforward to automate and can be useful for data quality monitoring even for companies who rely on third-party experimentation platforms.
We explain some challenges relating to list prices and more traditional modes of experimentation and we look into which testing capabilities we are building ...
The Experimentation Hub is central to our strategy, but it alone is not sufficient to scale experimentation across our organisation. We will describe the hyb...
I was interviewed for the 1,000 Experiments Club podcast.
I will draw a rough outline of the kind of culture that we are trying to build at Vista, and how we are taking the first steps towards that vision.
Abstract: Companies run A/B tests to accelerate innovation and make informed data-driven decisions. At Microsoft alone, over twenty thousand A/B tests are ra...
Three things I wanted to experience in my next job
we will examine in some detail why we consider research and experimentation to be the “peanut butter and jelly” of the customer insights world — two unique t...
FOMO made me do it
I was interviewed for the Diggin Travel podcast.
Kickstarting and keeping the A/B testing momentum
I was interviewed for the CRO Cafe podcast.
How I learnt to stop worrying and made the inbox-as-to-do-list approach work for me.
Abstract: Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies into an experimental laboratory. Despite...
Abstract: Accurately learning what delivers value to customers is difficult. Online Controlled Experiments (OCEs), aka A/B tests, are becoming a standard ope...
Mediation Analysis to Disentangle Direct and Indirect Effects
How we use online controlled experiments at Booking.com to release new features faster and more safely
Categorising Common Challenges
Read the full paper: Experimentation growth: Evolving trustworthy A/B testing capabilities in online software companies
Abstract: There is an extensive literature about online controlled experiments, both on the statistical methods available to analyze experiment results as we...
I was interviewed for the DataFramed podcast.
In October I’ll speak at Predictive Analytics World in London. Once again, I’ll be talking about Data Science.
My colleague Mats has an excellent piece on the topic of repeated significance testing on his blog.
In the next few weeks I’ll be giving two talks on the topic of Data Science at Xebicon and another event affiliated with Xebia. There is an abstract of my sp...
I am no longer a Corporate Ninja. As of a few weeks ago I can now call myself “Data Scientist at Booking.com”.
You could say fine dining is a bit of a hobby of mine; and as I’ve mentioned before, I’ve composed quite a few restaurant reviews over the years. I enjoy wri...
I’m not really interested in an exposition of your facts. I don’t very much care to learn about your reasons.
Note: I’ve tweeted about this before.
Oracle Data Mining (ODM) provides powerful data mining functionality as native SQL functions within the Oracle Database. This Oracle By Example Tutorial give...
The Wall Street Journal has an interesting article explaining how companies are starting to use (big) data to support their recruiting efforts. It provides a...
My girlfriend has been struggling with an interesting little problem lately. She was asked to determine the optimal distribution of medicine boxes and bottle...
Written by Arthur C. Clarke in 1976, Imperial Earth is set in faraway 2276.
Derek Jones posits that “success does not require understanding”.
Marketing catchphrases like “recommended by experts” (an appeal to authority), “world-renowned bestseller” (candidly claiming consensus) and “limited supply ...
It almost seems like everyone has their head in the cloud these days. And it’s not all just hot air and water vapor. Infrastructure as a service (IaaS), plat...
Note: Crossposting from the Oracle Real-Time Decisions Blog.
I’ll be chairing a implementation round-table session in Rome next month; involving both customers and partners. Sign up if you want to learn more about impl...
Without fail, a company will employ a recommendation engine for a purpose (nobody does this for fun, really). Often, that purpose is profit (or something alo...
Contrary to what it may seem like, I have been busy writing; just not for this particular blog.
James Taylor is spot-on.
Note: Crossposting from the Oracle Real-Time Decisions Blog.
There are two types of people in this world: Those who can extrapolate from incomplete data.
Facebook has recently discovered that beyond the uncanny valley of personalized marketing lies the bottomless pit of invasive identity misappropriation.
Note: Crossposting from the Oracle Real-Time Decisions Blog.
Dear [prospect.first_name],
Note: Crossposting from the Oracle Real-Time Decisions Blog.
Classical conditioning is underrated. Too many bad spy movies have taught us that ‘brainwashing’ is bad.
Analyzing website traffic can lead to unexpected insights. This incoming search term caught my eye.
In my mind, I write a blog post almost every day. But when I sit down to write, I am at a loss for words.
Note: Crossposting here from the Oracle Real-Time Decisions Blog where I have been invited to contribute.
I know it’s almost never necessary to use a semicolon, but I like semicolons; they make me feel smart.
Daring Fireball quotes some interesting research findings related to what Barry Schwartz dubbed The Paradox Of Choice.
After writing restaurant reviews for about a year, I’ve decided to apply my carefully whetted literary prowess to a new type of assessment: recommending peop...
I ran into the boss of the boss of the boss of the boss of my boss at London City Airport. He did not recognize me of course; there are quite a few Oracle co...
After consulting the Interwebs, I’ve added a little glitter to my website that will only be revealed to those that know the ways of the Konami code.
Gruber asserts.
Seth Godin on the topic of perception, memory and marketing concludes.
A few weeks ago, I showed a colleague1 my little visual demo of my favorite algorithm for estimating the number Pi. His immediate response was so obvious I a...
Computers are incredibly fast calculators. That makes them good at maths, but it does not make them smart. People that program computers have to invent cleve...
It seems that my little rant against the apparent lack of scientific rigor and the use of data to analyse performance in the world of advertising is nothing ...
3: Gratuitous use of Flash. It is not Adobe’s fault, it is your fault for using Flash for the most pathetic things mankind has known. Why? Because your...
It’s been a while. Too long, to be precise. I did not forget, I’ve simply been postponing writing anything for this blog for days on end. Not that I’ve been ...
After reading Performance Leadership by Frank Buytendijk it occurred to me that a good mission statement might not only be beneficial for organizations. The ...
The reason why a large percentage of business intelligence (BI) applications fail is not due to technology. To a large degree these applications fail beca...
I’ve started uploading some of the projects I’ve written about on this blog to Github.
After one of my previous posts about the snake project Casper had some questions.
Okay. So maybe I was jumping to conclusions a little early in the game.
Peter Norvig’s excellent essay Solving Every Sudoku Puzzle is a fascinating read, and I would highly recommend you read it (or at least scan it briefly) befo...
Speaking of intelligent critters, Wired has an excellent article on the topic.
Note: This post is based around a very old pet project of mine. I thought it worthwhile to describe this golden oldie some more detail and relate it to my...
Just like with the real coccyx, we should expect to see analogues to the legacy artifacts discussed in a previous post in other animals sharing recent evolut...
In a previous post I’ve explained how we can estimate the number Pi using Monte Carlo Methods (basically throwing darts at a circle). I personally think this...
You might not realize it, but your brain is running some pretty wonky legacy code. Stuff that worked fine back in the day, but is not very well suited for th...
I started a blog because I wanted to improve my writing skills; and thought I had something to say. You, as a reader can help me figure out how I am doing by...
I consider myself to be well organized. Both in my professional and personal life I use Remember the Milk for keeping track of almost everything I do using s...
I love reading. I love reading about statistics. I love reading about psychology. But most of all, I love reading quite a few books about statistics and psyc...
A representative of a large company (which shall remain unnamed) recently called on me for some advice. They had accidentally loaded polluted data into their...
Note: I’ve tweeted about this before.
Frank Buytendijk, in his book Performance Leadership, writes the following. Most people I asked, and most sources I referred to, define an organization si...
I’ve finally gotten ‘round to rooting my ‘old’ G1 phone and installing the Cyanogen firmware. I will not bore you with the technical details of flashing firm...
Because of the continuing lack of internet connectivity at my new address I’ve decided to move my website from my own server to the free space provided by my...
After posting my previous article I started wondering why my transactional data contained a disproportional amount of twos. Something must be distorting the ...
In a (not so) recent episode of Radio Lab (one of my favorite podcasts) I was introduced to Benford’s Law. I’d never heard of this phenomenon, which in hinds...
I’ve seen people using and building upon databases in lots of different ways; many of them surprisingly original. However, it seems to me that some people do...
At first, the term Business Intelligence (abbr. BI) had me a bit confused. Contrary to my initial interpretation, BI does not concern the mental acuity of co...
Note: I’ve tweeted about this before, but I feel the need to explain.
It crossed my mind before, but I never felt I had anything to say. I want that to change.
