In a recent review, which reads like a calm but clinical excoriation, Paap argues that many of these problems apply to research on the bilingualism advantage. “There’s a tendency to conduct multiple, small-sample studies that are underpowered,” he says. “That increases the likelihood of false positives. The problem is compounded by confirmation biases, or motivations to report only the studies that work.” (He stresses that he’s only talking about the purported cognitive advantages, not social or personal ones, of which there are patently many.)
For example, one group of researchers analyzed 104 abstracts on bilingualism that were presented at scientific conferences. They found that 68 percent of abstracts that found an executive-function advantage were eventually published in journals, compared to just 29 percent that found no advantage. This publication bias, a common problem in psychology and science as a whole, means that the evidence for the phenomenon seems stronger than it actually is.
But Paap doesn’t think much of the published evidence either. He found that a bilingual advantage only shows up in one in six tests of executive function, and mostly in small studies involving 30 or fewer volunteers. The largest studies, involving a hundred or more, all found negative results.
There are other problems, too. Many studies compare monolingual and bilingual people who vary in more ways than the number of languages they speak, including their nationality, educational level, socioeconomic background, immigrant status, and cultural traits. Any of these “confounding factors” could explain why bilinguals sometimes perform better in tests of attention or mental control, and very few studies satisfactorily account for them.
For that matter, Paap argues that it’s not clear what those tests are actually measuring. In his 2013 paper, he put his volunteers through four tests that are commonly used to study executive function. The scores on those tests should correlate with each other if they were actually measuring the same cognitive skill—but they didn’t. In some cases, the correlations were near zero.
“It was a hypothesis that many of us wanted to believe in,” he says. But he no longer does.
Paap’s review triggered 21 commentaries from other scientists, 15 of which were supportive. One, by Raymond Klein from Dalhousie University, is notable because he was a co-author on Bialystok’s seminal 2004 paper. “There were always aspects of the results that I was surprised with, mostly how big some of the effects were,” Klein tells me, “but I didn’t question whether those effects were right or not.”
He was writing positively about the bilingualism advantage until 2011, when he encouraged a student, Matt Hilchey, to review the studies that had accumulated in the previous seven years. “I think he was a little embarrassed,” Klein recalls. “He was meant to write a term paper for me, and I hadn’t suspected there’d be so much negative evidence.” Hilchey’s review highlighted the same problems as Paap’s—small studies, weak evidence, confounding factors—and came to the same conclusion.