Scale matters

In an altogether excellent piece on medical innovation, Tyler Cowen notes:

The NIH works as well as it does because the money is mostly protected from Congress. It is not a success which can easily be replicated. The more money is at stake, the more Congress wants to influence allocation. We should guard this feature of the system jealously and try to learn from it. If we can.

This is a seriously, seriously underrated factor in public policy analysis, and I include the libertarian variety. The fact that you can do something awesome with $15 million does not mean that you could do something super-awesome with $150 million. It may simply not be possible to broaden what you are doing very much before countervailing forces--such as congressional interference (Exhibit A: the goddamn Acela)--kick in.

Since we've been talking a lot recently about vouchers, education is one area where this is fairly easy to see. You get a pilot program: a curriculum, a teaching method, a high-intensity preschool program (such as the Perry program) for disadvantaged kids. You do a rigorous study of that pilot. It produces terrific results. Naturally, we should roll it out everywhere!

Not so fast. That pilot program has a huge administrative staff whose sole incentive is to ensure that it is meticulously carried out. In the real world, that curriculum will be put into place by an administrator whose priority list is crowded with everything from mollifying the latest lunatic on the school board, to ensuring that she gets out of town for a three day weekend with her new boyfriend who she really thinks may be The One.

That pilot program is staffed with a narrow band of extremely highly qualified teachers, sifted from the best the environment has to offer. In the real world, whoever happens to be standing in front of the classroom come September 5th has to do it, even if they flunked Remedial Math four times and only got this job because the school board needed a body.

That pilot program is rigidly policed for deviations from standard procedure, because deviations will kill the accuracy of the result. In the real world, tranquilizing the kid who just pulled a knife during study hall may take priority.

The pilot program is supported by a crack team that will move heaven and earth to ensure its completion; if funds are tight, they will not sleep until they have procured another grant. In the real world, it's probably less important than redecorating the teacher's lounge.

The pilot program has buy in from all participants; schools, teachers or students who don't like it, don't believe in it, or don't want it anyway, have already naturally dropped out of the sample. They will thus be striving to actually put it into place as closely as possible as described in the prospectus. In the real world 60% of everyone will think this is a moronic idea, and most of the rest will strenuously resent the intrusion on their autonomy.

Result: what worked beautifully in pilot will generally fail miserably in wider execution.

In business, these facts are summed up (over and over) with the dolorous mantra: didn't scale. In the public sector, that realization is still coming very hard.