There is a crisis in psychology. It’s not those rare cases of outright fraud, as when the social psychologist Diedrik Stapel simply made up the results of dozens of experiments and published them in top journals. The more serious problem—the one that keeps many of us up at night—has to do with the practices of honest and well-intentioned researchers.
Over the last several years, commentators have pointed out that much of psychologists’ standard operating procedure—our style of collecting data, analyzing our results, reporting our findings, and deciding what to submit to publication—is biased toward “false positives,” where random effects are reported as significant findings. Too many of us engage in “p-hacking,” for instance, where we rummage through our data looking for statistically significant findings, and then, with the most innocent of intentions, convince ourselves that these findings are precisely what we predicted in the first place.
It’s not surprising, then, that when other psychologists attempt to replicate published work—as was done recently with 100 studies, in a much-cited article in Science—most of the results don't hold up.
This is bad news. It means that when a new finding is published in one of our flagship journals, an informed psychologist can wonder about how many times the authors tried variants of their experiment before striking gold, what analyses they didn’t report, whether they crafted their hypotheses in respond to their findings, and whether they would get the same findings if they did the experiment again. I just read a study reporting that drinking sauerkraut juice makes you more likely to support extreme right-wing ideology. Maybe this is a robust and powerful finding, and maybe if you did the study again, you’d get the same result. But I’m entitled to be skeptical.