I’ve learnt about this new concept ‘survivorship bias’. It’s very interesting thing to be aware of so I thought I’ll share it here. This could also be very useful in your day-to-day life. It’s a simple but great concept about the human perception and thinking and how they impact statistics and their interpretations. Survivorship bias is well . . our bias towards the survivors. Here is the wikipedia definition:

Survivorship bias is the logical error of concentrating on the people or things that “survived” some process and ignoring those that didn’t. This can lead to false conclusions in several different ways. The survivors may literally be people, as in a medical study, or could be companies or research subjects or applicants for a job, or anything that must make it past some selection process to be considered further.

Survivorship bias can lead to overly optimistic beliefs because failures are ignored, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than being just lucky.

Wiki also goes to explain this with a lovely example.

If the three of the five students with the best college grades went to the same high school, that can lead one to believe that the high school must offer an excellent education. This could be true, but the question cannot be answered without looking at the grades of all the other students from that high school, not just the ones who “survived” the top-five selection process.

I thought this is a great concept for us to be aware. The fundamental point here is about how well the sample that we consider represents the overall population. It is a general tendency to ignore the failures and consider only the ‘survivors’ as our sample. Imagine a company opening about 100 funds of which over a period of time they close down about 30 of their worst performing funds. Now the balance 70 funds, which are the ‘survivors’ would certainly have good returns. Now the company can brag how their funds are outperforming the market. What we might miss to see is that these funds are the survivors who would naturally have a higher skewed average.

During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-air machine guns and German fighters. Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner’s station. Few bullets were found in the main cockpit or fuel tanks. The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that’s wrong. Planes with bullets in the cockpit or fuel tanks didn’t make it home; the bullet holes in returning planes were “found” in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.

In this fantastic example, the sample data for the research must have been what happened to the planes that were shot down. That’s what would help them in bringing back more people alive. The research on the returned planes and in particular, the decision to install heavy armour plates in the areas with maximum hit, is a classic display of survivorship bias.

While statistics is about interpreting the data available, what’s also important is to know about the data that is not available and the significance of the unknown data in the projections based on statistical methods. Also, the fact that there are failures which are being removed out of the system, makes any comparison to the past data potentially meaningless.

As you’re aware, in large organisations, on a periodic basis, there would employee feedback surveys. Assume that the survey shows that about 20% are terribly unhappy. Suppose these 20% staff leave the organisation, the newer survey tends to show a better result than last year’s one. The latest survey results are better because it considers the feedback of only the survivors and hence the result could be skewed.

Why does this survivorship bias exist? Can we avoid it? Traditionally, we are biased towards survivors or winners. We read stories, biographies and autobiographies of the winners and survivors to learn how they did it. There’s more to be learnt from the ones who did not survive. Studying the survivors alone could produce a skewed result. The population cannot be complete without taking into account the ones who did not survive.

The other day at lunch, one colleague of mine was saying that there are almost no bad actors in Hollywood (when compared to Indian movies), only for another colleague to quickly point out that ‘probably only those kind of movies don’t get released here’. That’s classic case of survivorship bias and subsequent realisation. Despite our understanding of statistical sampling, we tend to think that the sample of movies that’s released in India is the whole population of Hollywood movies made.

So how does this help us? Why would we have to be aware about this concept? Learning about the survivorship bias makes you less vulnerable to be fooled by this phenomenon. It helps you to ask the right questions. It helps you to put things in perspective. It helps you to read beyond the numbers and also remind you about the significance of the unknown data.

I’m very impressed with this concept. If we are not careful, this bias or a pitfall that we could fall into, very easily in our day to day life. As soon as I read this as a concept, I’m able to see this bias exists almost everywhere and that’s being exploited. Hope it helps you too, if not, at least this is a new thought to debate.