Is There Survivorship Bias in Index Performance? Seeking Alpha Edited

Before you accuse me of being doddy or repeating myself, let me just say this is a second post of a recent post which was published by Seeking Alpha.   The Seeking Alpha edited version is available below or you can follow the jump to the article on their site.  The latter of which has some rich commentary from other participants on Seeking Alpha.  Thanks for reading!

Is There Survivorship Bias in Index Performance?


In the performance analysis of hedge funds, survivability bias -- the logical error of concentrating on funds that "survived" some process and ignoring those that didn't -- can skew the performance results significantly for an investor or hedge fund of funds interested investing in hedge funds with similar strategies.




For instance, many weak funds are closed and merged into other funds to hide poor performance, i.e. GLG Partners (GLG) being bought by MAN, or Amaranth closing its doors for good, can augment a historical view and spin a positive bias on the results of the "Survivors." If you were to exclude GLG and Amaranth (granted, they were dissimilar strategies) in your measure of return performance in a given basket of hedge funds starting by including only those funds existing as of today in a five-year look back, you would be susceptible to creating a positive skew in your performance return numbers. More often than not, even sophisticated investors seeking the best performing hedge funds or mutual funds will inadvertently be relying on fund or peer performance data that is positively skewed mainly because it doesn't include the return performance of weaker or dead funds.



When hedge funds and mutual funds develop marketing materials for their funds they will often include a benchmark (S&P 500, MSCI Hedge Fund) and their return performance as compared to their peers (funds with a similar strategy). They will often utilize a universe of funds that exclude dead or merged funds, as this puts their returns in a better light. This can result in a discrepancy of up to 1.6% of additional return performance (Alpha).



Essentially, you will have a fund that has an 11% return with the bias, instead of a 9.4% return as calculated without the bias. When you are discussing amounts greater than $1MM this 1.6% is substantial. Since this bias can occur in many situations it is not inconceivable that this can also occur in indices, etf's or any defined basket of funds that trades or that can be invested in. It is also likely that positive survivorship bias has pushed indices like the DOW 30, S&P 500, NASDAQ 100 and Russell 1000 to an un-natural positive skew over time.



Survivability bias in effect plays into a lot of how we perceive the world and it can make a very real impact, positive or negative. Let's look at a couple of different examples.



Bullet holes: A brain teaser - During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-air machine guns and German fighters. Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner's station. Few bullets were found in the main cockpit or fuel tanks. The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that's wrong. Planes with bullets in the cockpit or fuel tanks didn't make it home; the bullet holes in returning planes were "found" in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.



This is a literal example of "survivor bias" — drawing conclusions only from data that is available or convenient and thus systematically biasing your results. Another great example of survivorship bias can also be found in business advice books. Here are some specific examples of survivor bias in business advice:



So far I've ("a smart bear") been asking rhetorically whether survivor bias might be severely skewing business advice. Steven Levitt (of Freakonomics fame) investigated this question directly.



He (Mr. Levitt) was reading Good to Great by Jim Collins, a book that analyzed eleven companies that were mediocre, but then transformed themselves into stock market sensations. A conclusion was that the common trait was a "culture of discipline." This book has sold many millions of copies, so it's a good example of popular writing on business advice. One of the eleven "great" companies was Fannie Mae (FNM), and Steven Levitt was reading this book just as Fannie was collapsing in financial disaster. Hmm, he thought, I wonder how those other "great" companies are doing. Turns out, had you invested in those eleven companies in 2001 (when the book came out), your portfolio would have underperformed the S&P 500! (Fannie Mae wasn't even the only case of total disaster — also extolled was the now-bankrupt Circuit City.) Why didn't these companies continue to succeed?



It turns out Jim started by combing through 1435 companies looking for good candidates for the book, and picked eleven. With such a large sample size he was bound to find companies that fit his criteria, however, that didn't mean that they were great due to his hypothesis. On top of that, Jim doesn't bother asking whether any of the 1424 other companies also displayed a "culture of discipline." Maybe that's something that many public companies have regardless of performance. Is this book an aberration? Nope, Steven investigated another business book from the 1980s — In Search of Excellence — and found the same effect.



Given the previously mentioned scenarios for survivability bias we can extend this to the process of selecting and or deselecting stocks that are listed in the major indices, e.g. DOW 30, S&P 500 and the Russell 1000. Specifically, in the process of rebalancing the indices it is the tendency for failed companies to be excluded from indices because they 1. No longer exist, 2. Their market capitalization has fallen or 3. Their industry is in decline (which likely caused the first two reasons); this is considered Type 1, survivor bias. Inherent in this type of bias is the error you make in just counting the survivors.



Another type of survivability bias, is associated with companies which are successful enough to be included, but because they have not met the criteria for inclusion until recently, their five-year look backs tend to include uncharacteristically high rates of return.



This is described as, "the error of inclusion prior to qualification" or Type 2 survivor bias. This can introduce abnormally high return data if you were to include a company which today was added to the index vs. a company than has been "qualified" and in the index for some time. I imagine ETF's as a group have a propensity for huge performance survivorship bias, but this is just a hunch, not something I am interested in determining, but you might.



Does this mean that indices like the S&P 500, The DOW 30 and the Russell 1000 are inherently flawed?



Frankly I don't know and it is difficult to determine. I am not advocating that the indices are flawed… well, OK they are flawed, but it's more a shortcoming than a handicap. Moreover the issue becomes how the underlying components of these indexes are viewed by people whose job it is to deconstruct the indices for a living. Think about this, a company gets dropped from an index. While a large amount of care is taken that the weighting of the indices isn't impacted negatively, a smaller amount of care is taken that a stock might perform too positively at the point of inclusion, but what of the next few years? With inclusion of the new next generation up and comer company, it is likely that it has been viewed as a leading contemporary of the future economy.



This suggests in essence an inherent upward bias in the indices. It's like changing out tired horses on the pony express for fresh legs. In December 2001, Enron was replaced in the S&P 500 with NVIDIA (NVDA which brought the S&P to include approximately 77 NASDAQ weighted stocks. NVIDIA a 21st century stock replacing a 20th Century also ran, shenanigans notwithstanding. As an offset to the previous example TYCO was also replaced, by Northeast Utilities (NU). So while there is a bias in the indices, it's not something to be running from. The propensity of the index though given the selection and de-selection process suggests that it is positively biased.



What are the possible ways that survivorship biases affect the indices?



Money managers, fund managers, investors and even Traders struggle with this issue of survivability bias because it can cause a real discrepancy between a thoroughly back tested trading model and the real life market. In mutual funds many well regarded fund managers believe that survivorship bias can also overstate a mutual fund's performance returns by more than 1.6%. A trader struggles with it when the universe of stocks they selected by measures of liquidity and market capitalization changes over time. This assumes that their universe stays static and the indices of course do not. The problem with indices relative to a static universe of stocks a trader is likely to select for their portfolio is outlined here in a white paper by Tick Data:



Universes where membership is based upon capitalization, such as the Russell and S&P indices, reward (include) companies whose stock prices have been outperforming, i.e. rising in relative ranking based on capitalization, and punish (remove) companies whose stock prices have fallen such that their market capitalization no longer qualifies for inclusion in the index.



For example, the 1050th company in market cap experiences relative outperformance versus existing members of the index and its market cap increases in rank to 990th. That stock then becomes a member of the index on the next index rebalancing date. In the meantime, an underperforming company that was a member of the index is crowded out as its market cap now falls below the 1000th largest. The outperforming stock is in and the underperforming stock is out. A trader that defines his/her universe on the day following such a theoretical event will test his/her trading strategy only on the outperforming company and will never see the impact of the underperforming stock on the strategy's results. This is survivorship bias.



However, it gets worse. Real-time practice begins to disconnect from simulation almost immediately. The next stock that rises up the ranks of capitalization to merit inclusion in the index will not be added to the universe. Again, I am assuming the universe, once defined, remains static. The underperforming company that was just crowded out of the index is not removed from the universe. As a result, in real time the trader is not trading the outperforming company, is trading the underperforming company, and both are in direct opposition to what was done in simulation.



Moreover, how can survivorship bias impact index performance?



Well consider this scenario which Tick Data provides in their white paper, which I borrow heavily from to make my point. Enron, Worldcom Global Crossing, and endless dot com blowups maintained substantial influence in the Russell 1000 during Tick Data five year test period, 1998 - 12/31/2003. However by virtue of defining the universe (RUS1000) as of 12/31/2003, these companies, and their negative downside performance had been excluded. As time and distance from these points of failure increased so did the positive skew in the Index data from which many traders made their assertions and recommendations for investment decisions.



This is inherently problematic on two levels. On one level it creates a false level of optimism when looking at the Russell 1000 Index as the companies included in the index on or by 2003 excludes a significant number of major deadbeats, and in the same vein the companies that were replacements to the dead beats most likely exhibited extraordinary growth in a short time period, Type 2 bias e.g. Carmax which was added to the index in December 2002 and since they had a relatively short existence their 5 year historical analysis include returns of 477% in 2001 and 66% in 2000 this gets added to the mix and causes people deconstructing the indices to their basic components to drastically overstate an indexes relative performance.



This further compounds the optimism as you now have a company included in the data and the five year look back feeds the current optimism about the future value of the index. This is why you can have some people convinced that stocks are undervalued and other(s) are convinced stocks are way overvalued. Leaving investors thoroughly confused and scratching their heads wondering whether to stay or go.



The second level is a bit more sinister as, when the "deconstructionists" forgo inclusion of the dead beats in their five-year look backs, they gloss over the amount of risk you take on, ignoring the very real possibility that a future bunch of drop outs like an Enron, Global Crossing etc. etc. can be modeled in your risk profile or their risk analysis. The end result is that it can't, and this creates a problem in modeling and assessing future risk appropriately, because you have sheltered your model from Enron/Lehman (LEHMQ.PK)/Bear risk. This is over simplified, but can provide a good context with which to put the 2003-2007 rally into context, led by Apple (AAPL), Google (GOOG), eBay (EBAY) and a stalwart of other 21st century companies. All of a sudden stocks got really undervalued because the dead beats were gone and new thoroughbreds were added. It's akin to having a market that resembles a narcissists' selective historical view of their own performance attributes.



So what does this mean right here and now?



Well much like the survivorship bias was likely skewed positively from 2003-2007, we have ascribed that notion to the 2009 rally, especially since December 2009 – March 2010 many bad performances fell off the horizon. What this means is that the "deconstructionists" and their chief prognosticators are likely to start getting bullish when the Bear Stearns, Lehman's etc. etc. are just specks in the rearview mirror, and the inclusion of a new batch of upstarts creates an open road off into the horizon. However, we need to learn more about the planes that were shot down, before we can move forward otherwise we will unnecessarily risk very possible repeat of 2008-9. Be a survivor.

Is There Survivorship Bias in Index Performance? Seeking Alpha Edited