Survivorship Bias or why you shouldn't let your cat jump off a building

Or use stories to make decisions. Or rely on incomplete data. Or think that becoming an Instagram celeb is easy.

Summary

• Survivorship Bias is a logical error when decisions are made based on incomplete data that's gone through the selection process. It affects all areas of human life, from business to personal life.

• It often affects us when we look at successful cases (companies, people, cats...) and assume that success tells the whole story. In contrast, there's usually a massive amount of opposite but not visible cases. Focusing on successful outliers doesn't take into account the base rates.

• We're most influenced by stories that present one or more examples of success and ignore the invisible failures.

Cats and Planes

Does this image look familiar? I mean the airplane on it, not the cat... If not, you're about to learn something new and exciting.

Long story short: Back in World War II, Americans were trying to make their planes more robust. They looked at the pattern of bullet holes on planes returning from combat. You can see it on the screen. And the typical thinking at that time was to add armor to the places that were hit the most.

But then a smart mind noticed that the planes that they could analyze were the ones that were coming back from the missions. And what they don't see are the ones that crashed... So what that means is that the surfaces that they don't see holes in are the most vulnerable parts of the aircraft that can cause it to crash. And they started reinforcing those areas instead.

The logical error of making decisions based on the data that went through the selection process in some form and ignoring the missing data points is called Survivorship bias (SB).

Unfortunately, people in any field of human activity are very susceptible to this type of bias, and you can find many examples of this bias on Wikipedia. One example that I particularly like is about... Cats.

A 1987 study showed that cats that fall from the 6th floor or higher are more likely to survive than those that fall from lower floors. The suggestion is that they reach a terminal velocity, and then it doesn't matter what height they fall from. In 1996, the newspaper column The Straight Dope suggested that another possible explanation for this phenomenon is survivorship bias. If a cat dies falling from a window, it's less likely to be taken to the vet than if it's injured. This is literally the survivorship bias. Don't let your cats fall out of windows randomly.

Stories Lie

Humans think in terms of stories. We tell them to others. There's even such a thing as Narrative Bias - a natural tendency for people to interpret information as part of a larger story or pattern, regardless of whether the facts support the overall narrative. Every time you hear a story about someone who achieved something great, remember that you're only hearing that story because of their success and not hearing all the other voices of people who failed while doing the same thing.

All those college dropouts who started unicorns - how many of them ended up as fast food workers? All those examples of successful companies that made millions - how many ended up in bankruptcy before the market even got to see their product? All those Instagram influencers that you see every day - how many of them you don’t see, because they are busy working real jobs to make a living?

Focusing on success stories doesn't take into account the base rates and gives a false impression that success is easier than it's. This leads to overconfidence - one of the most damaging traits for founders.

In my opinion, SB is an example of a number of biases people fall into when working with and failing to recognize incomplete data collection.

An excellent book by David J. Hand looks at dealing with different situations with missing information - Dark Data: Why What You Don't Know Matters. I highly recommend reading it, primarily if you work with systems full of uncertainty. It also shows some very concrete strategies for dealing with it.

Dark Data: Why What You Don't Know Matters

Summary

• Survivorship Bias is a logical error when decisions are made based on incomplete data that's gone through the selection process. It affects all areas of human life, from business to personal life.

• It often affects us when we look at successful cases (companies, people, cats...) and assume that success tells the whole story. In contrast, there's usually a massive amount of opposite but not visible cases. Focusing on successful outliers doesn't take into account the base rates.

• We're most influenced by stories that present one or more examples of success and ignore the invisible failures.