Why Do Science and Business Experiment More Than Government?
A new book by Jim Manzi argues that more randomized control trials of government programs could measurably improve public policy outcomes.
To understand the argument that entrepreneur Jim Manzi makes in Uncontrolled: The Surprising Payoff of Trial-and-Error For Business, Politics, and Society, it's useful to forget everything you believe about why freedom is important. Forget the Declaration of Independence, the tyrannical king to whom it was sent, and the notion that we possess natural rights to life, liberty and the pursuit of happiness. All of that is completely irrelevant to a less familiar, more utilitarian argument about why this country's citizens ought to be afforded freedom.
This argument holds that ignorance is inevitable in a world as complicated as ours -- that it's impossible for anyone to be certain of the best way forward at a given moment. But a free society permits individuals to try different ways of living. Successes are imitated. Customs develop around what works. The things that don't are abandoned or made taboo. And so long as there's always an ability to depart from custom or challenge taboos, important innovations keep happening. In this telling, the freedom to engage in trial-and-error is indispensable to progress.
These aren't new insights.
They're core to the classical liberal tradition and its skepticism about the ability of policymakers to innovate via central planning. If Uncontrolled were merely a restatement of the need for epistemic humility among wonks and legislators, interest in it might be confined to the right. The book is of broader interest, and may turn out to be important, because its author makes a compelling argument for an ideologically neutral method for improving policy, one that left and right might both plausibly embrace, even as it challenges both sides to rethink some of their reflexes.
That method is a specific kind of experimentation.
The first part of Uncontrolled is an enjoyable summary of how philosophers, scientists, medical researchers and corporations all discovered -- at different times and places -- that conducting lots of rigorous experiments is the most reliable way to figure out what works and what doesn't work. As Manzi takes pains to demonstrate, the gold standard in experimentation is the double-blind, randomized field trial with a control group. Fortunately, technology has made conducting these sorts of experiments more cost effective than ever.
But they're rare in public policy.
That's partly due to their limits. "The reason we have increasing trouble building compact and comprehensive predictive theories as we go from physics to biology to social science is the increasing complexity of the phenomena under investigation," Manzi writes, which also makes it "far harder to generalize the results of experiments."
For example:
We can run a clinical trial in Norfolk, Virginia, and conclude with tolerable reliability that "Vaccine X prevents disease Y." We can't conclude that if literacy program X works in Norfolk, then it will work everywhere. The real predictive rule is usually closer to something like "Literacy program X is effective for children in urban areas, and who have the following range of incomes and prior test scores, when the following alternatives are not available in the school district, and the teachers have the following qualifications, and overall economic conditions in the district are within the following range." And by the way, even this predictive rule stops working ten years from now, when different background conditions obtain in the society.
But that's no reason to give up. Government is just going to be a supplier of social welfare, education, policing, defense, medical research, and more. Thus the book's suggestion: conduct lots of rigorous experiments to test the effectiveness of every new policy that's implemented.
Make that rigor a reflexive part of government culture.
Says Manzi:
For the hypothetical literacy program described above, an experiment to test the program is not really a test of the program; it is a test of how well the program applies to a specific situation.
A brute-force approach to this problem would be to run not one experiment to evaluate whether this program works, but to run hundreds or thousands of experiments to evaluate the conditions under which it works. If it could be tested in a very large number of school districts, we might very well discover some useful approximation to the highly conditional rule that predicts its success. This is the opposite of elegant theory-building... But it might provide practically useful information.
Confident that its now plausible to do lots of large-scale, cost-effective policy experiments, Manzi wants the federal government to create an agency charged with running and refereeing them. And he wants states to enjoy greater freedom to experiment, if they agree to rigorously test the efficacy of everything they do, making the results transparent so best practices can spread.
His claim is that this can modestly improve policy at the margins.
There's a lot more to this book, which has been justly praised in every review I've read, partly because its the rare political book that goes out of its way to raise the most powerful objections to its arguments and to point out the limits of the reform program that it recommends. In a country where the left is sometimes reflexively confident in the ability of social scientists to model effective policy, and the right sometimes reacts to policy failures with reflexive do-nothing nihilism, a non-ideological method of distinguishing between good and bad policy is appealing. What I can't decide is if anyone in power has sufficient incentive to implement such a system.