Why Do Science and Business Experiment More Than Government?

A new book by Jim Manzi argues that more randomized control trials of government programs could measurably improve public policy outcomes.

beakers full Flickr user Amy Loves Yah.png
Flickr/AmyLovesYah

To understand the argument that entrepreneur Jim Manzi makes in Uncontrolled: The Surprising Payoff of Trial-and-Error For Business, Politics, and Society, it's useful to forget everything you believe about why freedom is important. Forget the Declaration of Independence, the tyrannical king to whom it was sent, and the notion that we possess natural rights to life, liberty and the pursuit of happiness. All of that is completely irrelevant to a less familiar, more utilitarian argument about why this country's citizens ought to be afforded freedom.

This argument holds that ignorance is inevitable in a world as complicated as ours -- that it's impossible for anyone to be certain of the best way forward at a given moment. But a free society permits individuals to try different ways of living. Successes are imitated. Customs develop around what works. The things that don't are abandoned or made taboo. And so long as there's always an ability to depart from custom or challenge taboos, important innovations keep happening. In this telling, the freedom to engage in trial-and-error is indispensable to progress.

These aren't new insights.

They're core to the classical liberal tradition and its skepticism about the ability of policymakers to innovate via central planning. If Uncontrolled were merely a restatement of the need for epistemic humility among wonks and legislators, interest in it might be confined to the right. The book is of broader interest, and may turn out to be important, because its author makes a compelling argument for an ideologically neutral method for improving policy, one that left and right might both plausibly embrace, even as it challenges both sides to rethink some of their reflexes.

That method is a specific kind of experimentation.

The first part of Uncontrolled is an enjoyable summary of how philosophers, scientists, medical researchers and corporations all discovered -- at different times and places -- that conducting lots of rigorous experiments is the most reliable way to figure out what works and what doesn't work. As Manzi takes pains to demonstrate, the gold standard in experimentation is the double-blind, randomized field trial with a control group. Fortunately, technology has made conducting these sorts of experiments more cost effective than ever.

But they're rare in public policy.

That's partly due to their limits. "The reason we have increasing trouble building compact and comprehensive predictive theories as we go from physics to biology to social science is the increasing complexity of the phenomena under investigation," Manzi writes, which also makes it "far harder to generalize the results of experiments."

For example:

We can run a clinical trial in Norfolk, Virginia, and conclude with tolerable reliability that "Vaccine X prevents disease Y." We can't conclude that if literacy program X works in Norfolk, then it will work everywhere. The real predictive rule is usually closer to something like "Literacy program X is effective for children in urban areas, and who have the following range of incomes and prior test scores, when the following alternatives are not available in the school district, and the teachers have the following qualifications, and overall economic conditions in the district are within the following range." And by the way, even this predictive rule stops working ten years from now, when different background conditions obtain in the society.

But that's no reason to give up. Government is just going to be a supplier of social welfare, education, policing, defense, medical research, and more. Thus the book's suggestion: conduct lots of rigorous experiments to test the effectiveness of every new policy that's implemented.

Presented by

Conor Friedersdorf is a staff writer at The Atlantic, where he focuses on politics and national affairs. He lives in Venice, California, and is the founding editor of The Best of Journalism, a newsletter devoted to exceptional nonfiction.

Google Street View, Transformed Into a Tiny Planet

A 360-degree tour of our world, made entirely from Google's panoramas

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

Google Street View, Transformed Into a Tiny Planet

A 360-degree tour of our world, made entirely from Google's panoramas

Video

The 86-Year-Old Farmer Who Won't Quit

A filmmaker returns to his hometown to profile the patriarch of a family farm

Video

Riding Unicycles in a Cave

"If you fall down and break your leg, there's no way out."

Video

Carrot: A Pitch-Perfect Satire of Tech

"It's not just a vegetable. It's what a vegetable should be."

Video

The Benefits of Living Alone on a Mountain

"You really have to love solitary time by yourself."

More in Politics

Just In