What Science Can Tell Us About Bad Science

In May, the journal Science retracted a much-talked-about study suggesting that gay canvassers might cause same-sex marriage opponents to change their opinion, after an independent statistical analysis revealed irregularities in its data [1]. The retraction joined a string of science scandals, ranging from Andrew Wakefield’s infamous study linking a childhood vaccine and autism to the allegations that Marc Hauser, once a star psychology professor at Harvard, fabricated data for research on animal cognition. By one estimate, from 2001 to 2010, the annual rate of retractions by academic journals increased by a factor of 11 (adjusting for increases in published literature, and excluding articles by repeat offenders) [2]. This surge raises an obvious question: Are retractions increasing because errors and other misdeeds are becoming more common, or because research is now scrutinized more closely? Helpfully, some scientists have taken to conducting studies of retracted studies, and their work sheds new light on the situation.

“Retractions are born of many mothers,” write Ivan Oransky and Adam Marcus, the co-founders of the blog Retraction Watch, which has logged thousands of retractions in the past five years. A study in the Proceedings of the National Academy of Sciences reviewed 2,047 retractions of biomedical and life-sciences articles and found that just 21.3 percent stemmed from straightforward error, while 67.4 percent resulted from misconduct, including fraud or suspected fraud (43.4 percent) and plagiarism (9.8 percent) [3].

Surveys of scientists have tried to gauge the extent of undiscovered misconduct. According to a 2009 meta-analysis of these surveys, about 2 percent of scientists admitted to having fabricated, falsified, or modified data or results at least once, and as many as a third confessed “a variety of other questionable research practices including ‘dropping data points based on a gut feeling,’ and ‘changing the design, methodology or results of a study in response to pressures from a funding source’ ” [4].

As for why these practices are so prevalent, many scientists blame increased competition for academic jobs and research funding, combined with a “publish or perish” culture. Because journals are more likely to accept studies reporting “positive” results (those that support, rather than refute, a hypothesis), researchers may have an incentive to “cook” or “mine” their data to generate a positive finding. Such publication bias is not in itself news—back in 1987, a study found that, compared with research trials that went unpublished, those that were published were three times as likely to have positive results [5]. But the bias does seem to be getting stronger: a more recent study of 4,600 research papers found that from 1990 to 2007, the proportion of positive results grew by 22 percent [6].

Of course, rising retraction rates also reflect the fact that scientists, journalists, and amateur watchdogs have begun scrutinizing research more closely. New data-analysis tools play a part, as does plagiarism-detecting software [7]. So do a number of ambitious recent efforts to replicate findings—with dismaying results. In 2012, a researcher then at the biotechnology company Amgen wrote in Nature that when his team tried to reproduce 53 landmark cancer studies, they could replicate just six [8]. And according to a news report in Nature, a project aiming to reproduce the findings of 100 psychology papers has managed to replicate results for only 39 of them (the project’s findings are still under peer review) [9].

This heightened scrutiny—the very scrutiny that likely contributed to the retractions surge in the first place—could help reverse the tide, by providing a powerful disincentive to bad behavior. As more scientific misconduct is exposed and shamed, researchers who were previously tempted to play fast and loose with their data may now think twice.

The Studies:

[1]McNutt, “Retraction of LaCour and Green” (Science, June 2015)

[2] Grieneisen and Zhang, “A Comprehensive Survey of Retracted Articles From the Scholarly Literature” (PLOS One, Oct. 2012)

[3] Fang et al., “Misconduct Accounts for the Majority of Retracted Scientific Publications” (Proceedings of the National Academy of Sciences, Oct. 2012)

[4] Fanelli, “How Many Scientists Fabricate and Falsify Research?” ( PLOS One, May 2009)

[5] Dickersin et al., “Publication Bias and Clinical Trials” (Controlled Clinical Trials, Dec. 1987)

[6] Fanelli, “Negative Results Are Disappearing From Most Disciplines and Countries” (Scientometrics, Sept. 2011)

[7] Giles, “Special Report: Taking on the Cheats” (Nature, May 2005)

[8] Begley and Ellis, “Drug Development: Raise Standards for Preclinical Cancer Research” (Nature, March 2012)

[9] Baker, “First Results From Psychology’s Largest Reproducibility Test” (Nature, April 2015)

About the Author

Bourree Lam is a former staff writer at The Atlantic. She was previously the editor of Freakonomics.com.

More comfortable online than out partying, post-Millennials are safer, physically, than adolescents have ever been. But they’re on the brink of a mental-health crisis.

One day last summer, around noon, I called Athena, a 13-year-old who lives in Houston, Texas. She answered her phone—she’s had an iPhone since she was 11—sounding as if she’d just woken up. We chatted about her favorite songs and TV shows, and I asked her what she likes to do with her friends. “We go to the mall,” she said. “Do your parents drop you off?,” I asked, recalling my own middle-school days, in the 1980s, when I’d enjoy a few parent-free hours shopping with my friends. “No—I go with my family,” she replied. “We’ll go with my mom and brothers and walk a little behind them. I just have to tell my mom where we’re going. I have to check in every hour or every 30 minutes.”

Those mall trips are infrequent—about once a month. More often, Athena and her friends spend time together on their phones, unchaperoned. Unlike the teens of my generation, who might have spent an evening tying up the family landline with gossip, they talk on Snapchat, the smartphone app that allows users to send pictures and videos that quickly disappear. They make sure to keep up their Snapstreaks, which show how many days in a row they have Snapchatted with each other. Sometimes they save screenshots of particularly ridiculous pictures of friends. “It’s good blackmail,” Athena said. (Because she’s a minor, I’m not using her real name.) She told me she’d spent most of the summer hanging out alone in her room with her phone. That’s just the way her generation is, she said. “We didn’t have a choice to know any life without iPads or iPhones. I think we like our phones more than we like actual people.”