Sketch December 2012

The Data Vigilante

Students aren’t the only ones cheating—some professors are, too. Uri Simonsohn is out to bust them.
John Cuneo

Uri Simonsohn, a research psychologist at the University of Pennsylvania’s Wharton School, did not set out to be a vigilante. His first step down that path came two years ago, at a dinner with some fellow social psychologists in St. Louis. The pisco sours were flowing, Simonsohn recently told me, as the scholars began to indiscreetly name and shame various “crazy findings we didn’t believe.” Social psychology—the subfield of psychology devoted to how social interaction affects human thought and action—routinely produces all sorts of findings that are, if not crazy, strongly counterintuitive. For example, one body of research focuses on how small, subtle changes—say, in a person’s environment or positioning—can have surprisingly large effects on their behavior. Idiosyncratic social-psychology findings like these are often picked up by the press and on Freakonomics-style blogs. But the crowd at the restaurant wasn’t buying some of the field’s more recent studies. Their skepticism helped convince Simonsohn that something in social psychology had gone horribly awry. “When you have scientific evidence,” he told me, “and you put that against your intuition, and you have so little trust in the scientific evidence that you side with your gut—something is broken.”

Simonsohn does not look like a vigilante—or, for that matter, like a business-school professor: at 37, in his jeans, T-shirt, and Keen-style water sandals, he might be mistaken for a grad student. And yet he is anything but laid-back. He is, on the contrary, seized by the conviction that science is beset by sloppy statistical maneuvering and, in some cases, outright fraud. He has therefore been moonlighting as a fraud-buster, developing techniques to help detect doctored data in other people’s research. Already, in the space of less than a year, he has blown up two colleagues’ careers. (In a third instance, he feels sure fraud occurred, but he hasn’t yet nailed down the case.) In so doing, he hopes to keep social psychology from falling into disrepute.

Simonsohn initially targeted not flagrant dishonesty, but loose methodology. In a paper called “False-Positive Psychology,” published in the prestigious journal Psychological Science, he and two colleagues—Leif Nelson, a professor at the University of California at Berkeley, and Wharton’s Joseph Simmons—showed that psychologists could all but guarantee an interesting research finding if they were creative enough with their statistics and procedures.

The three social psychologists set up a test experiment, then played by current academic methodologies and widely permissible statistical rules. By going on what amounted to a fishing expedition (that is, by recording many, many variables but reporting only the results that came out to their liking); by failing to establish in advance the number of human subjects in an experiment; and by analyzing the data as they went, so they could end the experiment when the results suited them, they produced a howler of a result, a truly absurd finding. They then ran a series of computer simulations using other experimental data to show that these methods could increase the odds of a false-positive result—a statistical fluke, basically—to nearly two-thirds.

Just as Simonsohn was thinking about how to follow up on the paper, he came across an article that seemed too good to be true. In it, Lawrence Sanna, a professor who’d recently moved from the University of North Carolina to the University of Michigan, claimed to have found that people with a physically high vantage point—a concert stage instead of an orchestra pit—feel and act more “pro-socially.” (He measured sociability partly by, of all things, someone’s willingness to force fellow research subjects to consume painfully spicy hot sauce.) The size of the effect Sanna reported was “out-of-this-world strong, gravity strong—just super-strong,” Simonsohn told me over Chinese food (heavy on the hot sauce) at a restaurant around the corner from his office. As he read the paper, something else struck him, too: the data didn’t seem to vary as widely as you’d expect real-world results to. Imagine a study that calculated male height: if the average man were 5-foot‑10, you wouldn’t expect that in every group of male subjects, the average man would always be precisely 5-foot-10. Yet this was exactly the sort of unlikely pattern Simonsohn detected in Sanna’s data.

Simonsohn launched an e-mail correspondence with Sanna and his co-authors; the co-authors later relayed his concerns to officials at the University of North Carolina, Sanna’s employer at the time of the study. Sanna, who could not be reached for comment, has since left Michigan. He has also retracted five of his articles, explaining that the data were “invalid,” and absolving his co-authors of any responsibility. (In a letter to the editor of Psychological Science, who had asked for more detail, Sanna mentioned “research errors” but added that he could say no more, “at the direction of legal counsel.”)

Presented by

Christopher Shea is a former author of The Boston Globe’s Brainiac blog, and until recently, he wrote the Week in Ideas column for The Wall Street Journal.

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well. Bestselling author Mark Bittman teaches James Hamblin the recipe that everyone is Googling.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well.

Video

Before Tinder, a Tree

Looking for your soulmate? Write a letter to the "Bridegroom's Oak" in Germany.

Video

The Health Benefits of Going Outside

People spend too much time indoors. One solution: ecotherapy.

Video

Where High Tech Meets the 1950s

Why did Green Bank, West Virginia, ban wireless signals? For science.

Video

Yes, Quidditch Is Real

How J.K. Rowling's magical sport spread from Hogwarts to college campuses

Video

Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

More in Technology

More back issues, Sept 1995 to present.

Just In