Timothy Lee's case for index-fund investing is obviously one that I'm familiar with, but I've never heard an adequate answer to my fundamental question: what's the difference between a manager of a fund and a manager of an index?
Indices, after all, don't create themselves. Employees in the Standard and Poor's division of McGraw-Hill are responsible for choosing which stocks to include in each of their numerous indices - and which to exclude. They have algorithms for making these selections which take into account a variety of factors (since the index is supposed to be both representative of the market and readily tradable), and no doubt there is some degree of human discretion involved as well. At a 20,000-foot level, the index manager is picking stocks: which ones to buy, and which to sell. Which is exactly what the manager of an equity mutual fund is doing. So what's the difference?
A huge difference is that the equity mutual fund manager has a financial incentive tied - in some measure - to performance. The people who select stocks for the S&P SmallCap Value Index are perceived to be performing a technical operation - selecting stocks that fit certain criteria such that the resulting index replicates the performance of a particular segment of the market. They are not trying to make money. And they are not compensated as if they are. By contrast, the manager of a small-cap value fund is supposed to be making money for her clients. If she does very well, she'll probably get a bigger bonus; moreover, she'll probably attract more capital, which will increase her compensation further; and, finally, her career prospects will be improved.
We'd expect that an incentive structure like this would drive improved performance. But, as Lee notes, the opposite is true: the managers who are compensated on performance do worse, for their clients, than the managers who are not. This should be a surprise.
Moreover, it's worth thinking about the free-riding problem with index funds. Consider what would happen if all funds invested in the stock market were indexed. What, in that case, would drive prices properly to reflect the prospects of the various companies? What moves markets to equilibrium is the action of market-makers seeking to find the price at which buyers and sellers balance out. If all buyers and sellers were index funds, all stocks would move up and down together, because the only activity would be generated by net inflows or outflows to or from index funds, plus the sudden jumps associated with adding or dropping a name from the index. In fact, if you take the example seriously, investing in index funds would be impossible if nobody invested except through such funds, because there would be no buyers to sell to, and no owners to buy from, except other index funds. Who will always be going in the same direction. Index funds, then, are only able to operate because of the existence of liquid markets, which are themselves dependent on large numbers of buyers and sellers operating from premises other than those of the indexers.