Published at KDD 2026 (32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining), Proceedings V.2, pp. 7533–7544. Co-authored with Ron Kohavi, Jakub Linowski, Andrey Andreev, Majed Dodin, and Joachim Furuseth.

Practitioners are often taught that they should not “peek” at experiment results and stop as soon as they see something they like. Peeking inflates false positive rates and undermines the guarantees a test is supposed to provide. But this advice, taken too literally, can leave people feeling paralysed when something appears to be going badly wrong mid-experiment. Should they wait for the pre-registered sample size to be reached? Or should they pull the plug?

I wrote the following little fable to help practitioners reason about that tension. No statisticians were harmed in the making of it.

A story about two statisticians, and possibly a bear

Two statisticians—Al-Khalil and Florence—went hiking in the woods together. Long walks away from civilisation are an excellent way to stimulate in-depth conversations about the nature of statistics while also staying physically fit and enjoying the nature of … nature.

After a few hours of walking and talking, dark stormy clouds start to form overhead. Al-Khalil and Florence begrudgingly interrupt their conversation to review their options for not getting soaking wet. They are too far away from the parking lot to make it back in time, and they forgot to bring rain coats or umbrellas.

Suddenly, Al-Khalil spots a small cave in the distance. He suggests they run over to see if they can shelter there. Florence agrees, and they start to jog.

Once they arrive just outside the cave, Al-Khalil turns to address his companion. “You know, Florence,” he says, “this is not Hampshire; this is the great outdoors.” The two agree that they have no idea what dangers could be lurking inside that cave. Al-Khalil volunteers to enter first as an experiment and promises to report his findings back to Florence.

Slowly, Al-Khalil starts walking into the cave. After about ten paces, he turns a corner and disappears out of view. Florence, waiting outside while rain starts to fall, can still hear Al-Khalil’s footsteps recede into the distance.

Then, suddenly, Florence hears something else. Something which sounds a lot like some big, furry mammal being roused from a deep slumber. Then there is a SLAP, a scream that sounds suspiciously like Al-Khalil, and finally a deafening silence emanating from the cave.

A few minutes pass while rain continues to fall and makes the only sound for miles around.

“Well then,” Florence finally says to herself, “that was not really a well designed experiment, now was it? We did not agree to a fixed runtime, so I have no idea how long I should keep waiting here. Also, sample size is really, really small (i.e. 1); and I am not sure Al-Khalil is really representative of the overall population of interest (i.e. me). We did not even clearly formulate any hypothesis we were going to test in the first place!”

With no clear hypothesis being tested, and no clear experiment design to execute against, an increasingly soaked Florence quickly concludes that she has no idea whether the cave contains a bear, a cougar, a dragon, or a snickering statistician having a laugh at her expense. The dangerous, the fantastical, and the innocuous are all still within the realm of possibility.

But even in the absence of a good experiment, reliable data, or a strong conclusion, statisticians are sometimes able to make a clear decision.

Quietly and ever so slowly, Florence starts backing away from the cave. After a few backward steps, she quickly turns around and starts to run into the pouring rain.

The moral of the story

Florence never learned what was in the cave. She has no data worth publishing, no p-value, no confidence interval. If a colleague back at the university asked her to prove there was a bear, she could not. And yet, running away was clearly the right call.

That is the tension worth sitting with. Statistical rigor is about drawing reliable conclusions from data. Risk management is about acting sensibly when the cost of being wrong is asymmetric. And some risks cannot be quantified at all—there is no metric for “probability of bear”—but they still have to be weighed. These two things usually align. When they don’t, protecting users should come first.

In practice, this comes up most often when an experiment appears to be causing serious harm. A checkout flow that seems to be losing revenue at an alarming rate. A feature that is triggering a spike in customer complaints. A change that has broken something you did not think it could break. In each case, the responsible thing to do is stop the experiment, even though doing so means giving up on a clean statistical result.

A few things are worth being clear about when you do this:

1. You are deviating from the plan on purpose. The whole point of a pre-registered testing plan is that you follow it. If you stop earlier than planned because you saw a scary number, you are no longer running the experiment you designed. That is fine, but you need to know—and say—that this is what happened.
2. You cannot claim the scary number was real. Florence can honestly say she ran away because she was afraid there might have been a bear in the cave. She cannot claim there was a bear. In the same way, an aborted experiment tells you that you were worried enough to stop; it does not tell you that the harm you feared was genuine.
3. Document the abort clearly. Mark the experiment as aborted, not as concluded. Explain in writing why you stopped. Future colleagues (including future you) need to be able to tell the difference between “we tested this and it did not work” and “we got spooked and ran away”.
4. Sequential tests solve peeking, not panic. Sequential testing methods let you look at results as they come in and stop as soon as the pre-registered criterion is met, without inflating error rates. This makes them a good option for monitoring, though fixed-horizon tests generally have more power and remain a reasonable choice for the primary experiment target. Either way, a sequential test only gives you valid inference about the metrics and stopping rule you designed for. If you abort because of something the test was not built to detect—a spike in complaints, an error rate you weren’t tracking, a gut feeling that something is wrong—you are still stepping outside the plan. The statistics on the metrics you did register are fine; the decision to walk away is a judgement call that lives outside the test.

Florence’s decision can also be automated. At Booking.com we built a circuit breaker that watches every new feature release for the first few minutes and aborts it automatically when severe user-facing errors spike. It deliberately cuts statistical corners—looser correction for multiple testing, approximate deduplication—in exchange for stopping bad releases within seconds rather than hours. The tradeoff is the same one Florence made outside the cave: when severe harm is suspected, speed of response can be worth more than waiting for a clean result. Building that kind of automated safety net is a good way to make the “better safe than significant” judgement explicit and repeatable, rather than leaving it to whoever happens to be watching the dashboard.

Why a story?

I like this story because it separates two things that practitioners often conflate: the quality of the evidence and the quality of the decision. Florence has terrible evidence. Her decision is excellent. Once you have seen that gap clearly in a story about a cave, it is much easier to see it in a dashboard about a checkout flow.

It also gives teams a shared shorthand. “Are we sure there is a bear, or are we just spooked?” is a surprisingly useful question to ask in a war room at 2am. It reminds everyone that stopping is allowed, that stopping is sometimes correct, and that stopping honestly is very different from pretending the experiment concluded.

A good practitioner knows when to trust the procedure and when to override it. Blindly following the plan while the cave growls at you is not rigor; it is just stubbornness with a p-value attached.

Type-I and type-II errors are a good example of a concept practitioners learn the definitions of, and then promptly confuse under pressure. I wrote the following little fable to help. It is a retelling of the classic “The Boy Who Cried Wolf”.

A story about a boy and his sheep

Once upon a time there was a boy named Aesop. Aesop lived in a small town in the mountains with his mother, several other villagers, and their flocks of sheep.

One day, the villagers heard a rumour that a fearsome wolf named The Alternative Hypothesis was spotted in the area. They worried that one of their sheep might be eaten, so they decided that Aesop should stand watch while the sheep were grazing in the mountains.

“Aesop,” the villagers told him “your job will be to guard the sheep”. The boy was then handed the ceremonial Horn Of Significance. “When you blow this horn, we will all come rush up the mountain” he was told. “Please use it only to warn us when the real wolf appears, because we do not want to come rush up the mountain in error.”

Aesop begrudgingly accepted the horn and his new role.

Early the next day, Aesop trudged up the mountain together with the sheep for his first day on the job. Initially, the work seemed easy enough; the sheep grazed while Aesop sat quietly in the shade of a large tree. But after a few hours of watching grass grow, the boy grew increasingly bored.

“What if,” Aesop wondered out loud “I had some friends to play with?” The boy stared at the horn in his hands. His friends were in the village below, far away. Suddenly an idea occurred to him. “What if,” he mused “I could use this horn to bring all my friends up here, so I would no longer be alone?”

No sooner said than done. Aesop blew with all his might and the significant blare of the Horn of Significance filled the valley. Within minutes, all the villagers came running up the mountain hoping to chase away the wolf named The Alternative Hypothesis and save their sheep. However, upon arrival they found only a very bored boy.

“Oh, Aesop!” the villagers exclaimed. “How you have fooled us. We heard the significant signal and concluded that there must truly be The Alternative Hypothesis, but now we see the error we have made was one of the First Kind.” Aesop was given a stern talking-to, no supper, and sent to bed early.

Early on the second day of work, the boy trudged up the mountain together with the sheep. Again the sheep grazed while he watched grass grow. The weather was hot and the job tedious. Soon Aesop grew sleepy from counting sheep. As he sat beneath the tree dozing, that fearsome wolf named The Alternative Hypothesis crept up on Aesop from behind. With one swift bite, the wolf gobbled up the boy whole.

Down in the valley, the villagers waited for the signal. They perked their ears trying to hear the positive sounds of the Horn Of Significance, but all remained quiet. The boy in the belly of the wolf could not blow the horn; the villagers could not hear a sound that was not made.

Later that evening, when Aesop and the sheep failed to return home, the villagers once again rushed up the mountain. There, they discovered the remains of their sheep and the discarded and unused Horn Of Significance.

“Oh, Aesop!”, the villagers lamented. “How we have been fooled again. We failed to hear the significant signal and concluded that The Alternative Hypothesis was not amongst us, but now we see the error we have made was one of the Second Kind.”

The moral of the story

The villagers committed both a type-I and a type-II error, in that order.

1. First they heard the boy cry (i.e. got a significant result) and falsely concluded that there was a wolf. This is a type-I error, or error of the First Kind, also known as a false positive.
2. Then they heard no cry (i.e. got an insignificant result) and falsely concluded that there was no wolf. This is a type-II error, or error of the Second Kind, also known as a false negative.

A small terminology trap worth flagging: the words “positive” and “negative” here refer to the result of the significance test (significant: yes or no), not to the direction of the observed effect. A result can be both negative (e.g. we observe fewer visitors click a link) and simultaneously a false positive (e.g. the observed difference is significant by chance, while in reality there is no underlying effect).

The deeper point is that we cannot fault a young boy for being bored; rather, we should blame the villagers for relying on an error-prone methodology. Type-I and type-II errors are properties of a procedure, not failures of the analyst. Because no method is perfect, we must always be mindful of potential error rates when making decisions. We should seek methods with controlled error rates. For example, sending two boys instead of one provides much-needed redundancy; a single experiment can be a fluke, but two measurements are better than one.

Why a story?

I have used this story in workshops and conversations for years. People who could not, the day before, reliably tell you which of type-I and type-II is the false positive will happily explain, weeks later, what The Alternative Hypothesis is and why we should not trust the Horn Of Significance. The story sticks where the definitions slid off.

It also gives us shared shorthand. “Don’t be the villagers” is a faster, kinder way to push back on a colleague’s overconfident reading of a p-value than re-deriving the definition of significance. And because the mistake belongs to the villagers and their methodology—not to anyone in the room—the story makes it easier to discuss the error without anyone feeling stupid.

That, more than the mnemonic itself, is why I keep telling it.

Published at SEAA 2025 (Euromicro Conference on Software Engineering and Advanced Applications), Lecture Notes in Computer Science, vol 16083, Springer. Co-authored with Nils Stotz, Ben Labay, and Paul Drews.