If you took to heart the recent cover story in The Economist, “How Science Goes Wrong,” you might be tempted throw your hands up and stop reading about scientific research entirely. The piece describes how scientists often fail to reproduce some of the most frequently cited findings in their fields, calling their conclusions into question. Science writers have also come under fire recently, most notably Malcolm Gladwell, who according to critics in the The Atlantic, The Wall Street Journal, and Slate, among others, cherry-picks research to fit his thesis and hangs major arguments on poorly replicated studies in his latest book, David and Goliath.
These rebukes are only the latest in what has become a torrent of criticism of the way scientific research is carried out and reported. The catalyst arguably was a paper by the epidemiologist John Ioannidis, provocatively titled “Why Most Research Findings are False,” that got a lot of attention from the popular press, including a 2010 cover story in The Atlantic by David Freedman. In a recent piece in the Columbia Journalism Review, Freedman also blamed science journalism for “a failure … to scrutinize the research it covers.”
Where are the readers in this discussion? Yes, scientists should do high-quality work and journalists should report it responsibly, but readers should be discerning and thoughtful information consumers. They can’t expect science writing to provide simple answers to complex questions; in fact, they should be skeptical of any piece that claims to do so. As University of Virginia psychology professor Brian Nosek told me, "Science reporting is not purveying the facts, it's purveying the discovery process, the adventure into the unknown."
Getting the most out of science writing takes work, but it’s vital, and similar to the attention we devote to consuming other products: We check the labels on food packages at the supermarket. We pore over online reviews before making even minor purchases. We should put the same care into the way we absorb scientific information, which has the power to shape the way we live.
To be a thoughtful reader, there are a few questions you should ask yourself whenever you read a popular science piece. I’ll use Hanna Rosin’s 2009 story in The Atlantic, about the evidence for and against the benefits of breastfeeding, as an example, because it addresses these questions especially well.
What’s the Big Picture?
A lot of science writing focuses on single studies. But each study is only a piece of the puzzle; look for writing that provides context. Before delving into the research on breastfeeding in her piece, Rosin provides historical background that helps explain why people have such strong feelings about breastfeeding and formula. She describes, for example, how some of the concern about formula stemmed from an international scandal in the 1970s, in which babies who were formula-fed in South America and Africa were more likely to die than those who were breastfed. It turned out that this was because mothers were using contaminated water or rationing formula because it was so expensive. Still, she writes, “the whole episode turned breast-feeding advocates and formula makers into Crips and Bloods, and introduced the take-no-prisoners turf war between them that continues to this day.”
Her review of the breastfeeding research is similarly nuanced. The article’s title, “The Case Against Breastfeeding,” suggests a diatribe, but the piece itself provides an even-handed review of the literature, describing studies that have found evidence for the benefits of breastfeeding and others that have not. In this way, she builds toward her overall conclusion: that claims about the benefits of breastfeeding have been overstated.
Even a good study has limitations and weaknesses. Usually the researchers are direct about these in academic papers, and do a decent job of explaining how they might detract from the study’s conclusions. Look for science writers to be similarly frank. Examples of weaknesses include: samples that are either very small (which makes it difficult to find a statistically significant difference) or very large (which means tiny, basically meaningless effects might still be statistically significant), or are somehow unrepresentative of the population they’re trying to understand (studying animals to learn about human diseases, for example).
Also look for a popular science piece to identify methodological weaknesses that might have undermined the study’s findings. In her piece, Rosin points out a “glaring flaw” with most research on breastfeeding: mothers who choose to breastfeed probably differ in many ways—income, education level, race—from those who don’t, and any of these factors could influence a child’s development. Even cleverly designed sibling studies, which compare mothers who fed their children differently—say, breastfeeding the first child but using formula for subsequent children—can’t completely address these confounds, Rosin points out. A mother might treat her children differently in ways other than feeding—for example, lavishing more attention on her first child than on her second and third. By pointing out the weaknesses in some of the breastfeeding studies, Rosin helps explain the contradictory literature: Studies that found evidence for benefits tended to be those that did not account for these potential confounds, whereas those that found little or no evidence for breastfeeding’s benefits were usually well-designed and controlled.