Explaining the Rise in Science's Errors
Why there are now more retractions in scientific journals
.A lot more science findings have been found out to be wrong recently, and no one is sure why. Yesterday, The Wall Street Journal quantified the trend: since 2001, there's been a 15-fold increase in retractions in scientific journals. That's a big deal, as those peer-reviewed studies form the basis for nearly all science reporting in the news. We wouldn't be writing posts noting how you can live longer, lose weight and avoid brain shrinkage without them. And while The Journal notes that the rise in noticed errors could just mean that science journals are getting better at catching mistakes, others are taking a dimmer view of the findings:
- The peer-review process is broken As Psych Central's succinctly explains, "peer-review is the process where journals vet incoming scientific articles by reviewers." That filtering process, he argues, is broken because researchers seem adverse to replicating the findings of other researchers, which may lead to more errors escaping notice:
[T]he foundation of our science — peer-review publication — is increasingly suffering from problems that make it tougher and tougher to not take anything read in a journal with a grain of salt. The scientific process has a solution to this problem, of course — it’s called replication by other, independent researchers of one researcher’s initial findings. However, in this fast-moving world, few people wait for replication any longer, and are quite happy to trumpet findings from tiny studies done on a few undergraduate college students.
- Why it's broken Scientists don't want to replicate each others' findings because they don't tend to get noticed that way. That was one of the theories laid out by David Freedman in an Atlantic magazine article last year. Simply put: "coming up with eye-catching theories is relatively easy, getting reality to bear them out is another matter." He describes this hypothetical scenario:
Imagine, though, that five different research teams test an interesting theory that’s making the rounds, and four of the groups correctly prove the idea false, while the one less cautious group incorrectly "proves" it true through some combination of error, fluke, and clever selection of data. Guess whose findings your doctor ends up reading about in the journal, and you end up hearing about on the evening news?...[S]imply re-proving someone else’s results is unlikely to get you published...
As we can attest, counter-intuitive--and sometimes flat out bizarre--findings are what gets attention. Unsurprisingly, when those widely-circulated studies get retracted "it can be hard to make its effects go away," says one specialist to The Journal.
- Why it's hard for faulty studies to be fully debunked Once a controversial, groundbreaking study takes hold in the media, it can develop a life of its own. One of the most notable examples of this was Andrew Wakefield's paper in The Lancet which claimed a link between a measles vaccine and autism. By the time it was finally retracted, popular perception of the vaccine had shifted due to the very controversial claims.
- And when papers are retracted there's a transparency problem After The Journal finding was published, Ivan Oransky the author of the blog Retraction Watch (which makes it a mission to note scientific journal errors) gave this explanation to journalist Ed Silverman about why more errors are being found:
"It’s unclear whether the actual amount of misconduct and legitimate error has grown; it may just be that we’re picking up on more of it," [Oransky says]. "What makes it difficult to tell is a problem we often see at Retraction Watch: Opaque and unhelpful retraction notices saying only ‘this study was withdrawn by the authors.’ How does that make for transparent science? We think journals can do a lot better, by demanding that authors and institutions come clean about what went wrong."