“Reading,” according to the writer Joyce Carol Oates, “is the sole means by which we slip, involuntarily, often helplessly, into another’s skin.” The idea that literature orients readers to the thoughts and feelings of others goes back at least as far as Aristotle, but only recently have psychologists tried testing it in the lab.

In 2013, a widely publicized study in the journal Science by David Kidd and Emmanuele Castano of the New School suggested that reading “literary” short stories immediately improved participants’ abilities to read the facial expressions, and thus the emotional states, of other people. Several media outlets (The Atlantic included) ran with the idea, embellishing it with headlines like For Better Social Skills, Scientists Recommend a Little Chekhov. Now, a recent study co-authored by Thalia Goldstein of Pace University in The Journal of Personality and Social Psychology calls those results into question.

The original experiments called for participants to read one of six texts and then take the Reading the Mind in the Eyes Test (RMET), in which people are made to look at photographs of actors’ eyes and select one of four states of mind the picture conveys. A raised brow could be a sign of panic, or a glaring eye a signal of anger. The number of correct answers is meant to measure theory of mind—the ability to attribute “the full range of mental states” that motivate action.

“It’s a poor tool,” Rebecca Saxe, an MIT cognitive neuroscientist who studies theory of mind, said. But “it is the only measure that we currently have ... which is why everybody uses it.” And no matter what the RMET’s weaknesses, both studies used it, and so the fact that they found different results is significant, Saxe said.

Though Goldstein’s group didn’t replicate the more famous finding of the original study, they did find a common result: People who were lifelong readers of fiction (measured by the number of author names they could recognize) had significantly higher scores on the RMET. “But this is a correlation”—not a causation!—“and it falls under all the caveats that we need to put on a correlational finding,” Goldstein added. A lifetime of reading might make people better at imagining other people’s thoughts and emotions, or those who are more in tune with other people’s states of minds might be drawn to reading fiction in the first place. Or, a completely unrelated variable might explain the correlation.

Along with raising objections from the authors of the original study, this newer paper is a reminder of the way reporting can reshape scientific ideas. Most popular coverage of the original study talked about this work in terms of  “empathy,” but the researchers themselves say that’s not quite the same thing as theory of mind. “The terminology here is definitely a little mushy,” Goldstein told me. “People in the field would be the first to admit that.” According to Goldstein, the finer distinction between theory of mind and ‘cognitive empathy’—the capacity to imagine rather than share another person’s feelings—is another fuzzy point in the field.  

When I spoke to Kidd and Castano, they told me that they believe Goldstein’s group failed to weed out research subjects who spent too little time reading each story from their study. Because participants took an online survey through Amazon’s Mechanical Turk, there is the risk that some simply clicked through the stories or kept the window open on their screen rather than sitting through the experiment, they said. (Goldstein declined to respond to a request for the reading-time data of her group’s participants). Still, several of the authors on Goldstein’s study came from other labs that independently found the same negative finding, and weren’t planning on publishing it until they found out Goldstein’s group was submitting their study for publication. As a result, their group brings a much larger sample size to the table.

This body of research runs into an age-old stumbling block that continues to dog those who study literature: What—if anything—makes a text “literary?” The “literary” stories in Kidd and Castano’s experiment were cobbled together from the Pen/O. Henry Prize stories as well as finalists for the National Book Award. Goldstein’s group used the same stories in their experiment (in order to replicate the original study), noting that the line between literary and popular fiction is “fuzzy and dynamic.”

“It's a hornet's nest, that question,” said Arnold Weinstein, a professor of comparative literature at Brown University. What gets to be called “literary” changes over time along with shifting ideas of social and political values, he said. Weinstein pointed to the controversy around Donna Tartt’s novel The Goldfinch as an example. After it won the Pulitzer Prize for Fiction, many critics complained the book wasn’t “serious” enough to merit the award.

It’s still an open question why psychologists, the media, and laypeople alike are so interested in the possible benefits of reading fiction. As Weinstein said, those both in and outside of the humanities have ascribed moral benefits to literature and art as “a rescue operation” for these disciplines at a time when their worth is under scrutiny. It’s hard not to see arguments that literature might make people more empathetic, more moral, or more socially adept as a corrective to the perceived lack of “return on investment” when it comes to the arts.

“I don’t hope or believe that social psychology is needed to justify the humanities,” Kidd told me. But in a culture where science is sometimes treated with more gravity than the humanities, this research can be used to do exactly that. In a recent piece for The Atlantic, a high school English teacher leans on psychological studies of a similar strain to strengthen his argument that students are emotionally enriched by reading fiction.

Even if literature fostered a stronger sense of theory of mind, or even empathy, Weinstein remains skeptical of what that would mean: “People used to comment that the people who ran the concentration camps probably knew Goethe and Schiller by heart,” he told me. While literature does produce “a kind of empathy,” he says, “the social uses and ramifications of that are extremely open to question.”

As the psychologist Paul Bloom writes, the benefits of empathy are sometimes seen as “too obvious for justification,” too easily conflated with the ideas of compassion, morality, and kindness. To Bloom, empathy is instead an emotional resource that has its right place and time—whether it does any good or fills a need depends on the context. Among the many calls to empathy after the election, some have aptly pointed out that empathy (both by psychology’s definition, along with the other values of the high-road that get attached to the word) is a two-way street. Calling on one side to empathize with the other, while probably not a bad thing, is just one example of the limits of relying on an overly idealized vision of what empathy can achieve.  

There are many things literature can ‘do.’ It can give voice to perspectives that are often silenced, offer refuge, and allow readers “try on” the lives of others. But this is slow work that depends so much on social circumstance. No reader, writer, or book exists in a vacuum. Trying to reduce the process of reading to a snapshot task leaves out some of this context.

Perhaps to really understand what happens in the messy, intimate process of reading, looking at individual relationships between readers and stories may be more worthwhile than examining literature as a generalizable stimulus. While psychological research may someday offer another enriching perspective on reading fiction—Goldstein mentioned the possibility of looking at how readers respond to texts over time—these relationships may be too nuanced, or have too many variables, to be fully described by studies like this, let alone the popular coverage that recasts them.