Explanations I do find plausible, or maybe plausible:
1. The sums are so large and the stakes so huge that it doesn't make any sense to shop on price This is what I focused on in my article. Think of an IPO as the biggest, most expensive wedding ever planned, and you can perhaps grasp some of the price insensitivity. Only huge deals with very powerful managements like, oh, the US Treasury, can afford to drive a hard bargain.
2. Regulatory barriers Compliance costs to being a big investment bank have risen. It seems pretty clear that the costs of being regulated have increased on both the public company and the bank sides of the transaction over the last decade. Whether or not you support this, compliance costs form a barrier to entry that may be keeping smaller players from competing.
3. Investment banking scale matters As the fees have eroded in things like executing stock trades, these things increasingly become basically loss leaders for IPOs, M&A and the prop desk. Without mega capital and deal flow, you can't maintain these operations at the scale needed to maximize the IPO returns.
4. The losses on the fees are too small for any one person to care This is a special case of number 1. 4% of an IPO is a lot of money in absolute terms. But to whom is it a lot of money? Not to the management, who are not going to risk millions over 4%. Not to the stockholders, who do not care that much whether they pay $400 or $416 for their shares. Maybe to the VC owners.
5. Auctions haven't worked too well. They're the natural substitute for the underwriting model, but so far, they've disappointed, for reasons Tyler Cowen explored recently. (Read the comments). Contra Pearlstein, this does not seem to be simply a case of Wall Street snookering its clients (many of whom are very savvy VC guys who used to be on Wall Street). The uncertainty surrounding an auction seems to be off-putting, for a lot of the same reasons that you don't want to walk into Target and start haggling with the manager over the price of a sweater.
I welcome other suggestions from readers.
This does not, mind you, mean I agree with Steve Pearlstein that adding competition to the IPO process is somehow going to bring down banker comp to a level he finds less offensive. You need only look to hedge funds to be disabused of that fantasy. Though it varies by year which departments deliver the big wins, Goldman's profits are as likely to come off the bond desk as they are from the folks doing IPOs or mergers. Hedge fund managers rake it in without doing any IPOs at all.
This even though most financial markets are ruthlessly efficient--not in the sense of being right, but in the sense of making sure that large price disparities (aka arbitrage opportunities or "free money") do not persist. That this is true over so many different areas leads me to lean heavily on the "taking a thimble out of a gigantic river of money" explanation. Which suggests that it will probably persist for quite some time. Individual markets will become more and more efficient as the assets are better understood--stocks used to be risky and complicated, and now they're mundane. But finance as a whole, like Hollywood and professional sports, will continue to offer many opportunities to become obscenely wealthy.