How Comments Shape Perceptions of Sites' Quality—and Affect Traffic

Even if you don't realize it, unmoderated comments change the way you think about what you read.

There’s a game I like to play sometimes. It’s called “How many Internet comments do I have to read until I lose faith in humanity?” All too often, the answer is: one comment.

From The Atlantic to Yahoo to YouTube, online comments are often ignorant, racist, sexist, threatening, or otherwise worthless. But you knew that already. There’s plenty of anti-comment sentiment on the web—some humorous, some more scholarly—and despite the hopes of media democratizers, there’s now widespread agreement that Internet comments are terrible. "Even in places with smart, thoughtful readers, the comment sections tend to be more like lists of unconnected ideas than genuine conversations," The Atlantic's Rebecca Rosen wrote in 2011. Some publications, like Popular Science, have given up on comment sections all together.

A couple of weeks ago, National Journal changed its comments policy, opting to eliminate comments on most stories as a way to stem the flood of abuse that appeared on the site. Naturally the comment-section reaction to that announcement helped reinforce the reason editors said comments had to go in the first place.

For all the boycott threats and comparisons to Hitler, though… the site seems to be doing better now. If anything, user engagement has increased since the comment policy changed. Pages views per visit increased by more than 10 percent. Page views per unique visitor increased 14 percent. Return visits climbed by more than 20 percent. Visits of only a single page decreased, while visits of two pages or more increased by almost 20 percent.

What happened here?

One theory: By cutting out comments, the site is better able to draw attention to its most deserving content—the articles themselves.

This intrigued me because I found it somewhat counterintuitive. I supported removing comments not because I thought traffic would spike but because it seemed a way to better preserve civil discourse; I assumed we’d lose some rubberneckers who gathered around the train-wreck comment section, but it seemed like a worthwhile trade. Yet the fact that traffic actually improved suggests that sites are better off without comments—or at least better off without unmoderated ones. That's a lesson that other news organizations are learning. As Nieman Lab wrote last month, if news organizations aren't moderating their comment sections, they can't really expect them to foster quality discussion.

But what about the many sites that opt for a less hands-on approach? Plenty of journalists will tell you that they not only don't reply to commenters, but that they don't even read the comments to begin with. An ignored comment section can't be all that harmful, right?

To find out, I ran a quick study using respondents from Amazon’s crowdsourcing platform Mechanical Turk. I asked 100 Americans to read a snippet of a National Journal article from late April. Half of them saw the article alone. The other half saw the article along with a representative sample of actual comments (user accounts redacted) on that article. In both groups, respondents were asked to read the article—the existence of comments was never acknowledged.

After reading, respondents took a short survey. The first two questions were designed simply to verify they’d read the article. This part was easy. All respondents needed to do was identify that the article was about Cliven Bundy, rather than Donald Sterling, Rush Limbaugh, or Barack Obama.

The remaining three questions, however, measured respondent perceptions of the article—and asked for the person's current mood on a five-point scale. The article was the same for all readers, so if comments had no impact, both respondents who saw comments and those who didn't would evaluate the article and their current mood the same way.

That was not the case.

Respondents who saw comments evaluated the article as being of lower quality—an 8 percent difference. In other words, authors are judged not just by what they write, but by how people respond. The presence of comments did not make a statistically significant difference in a person's likelihood to read more content by the same author, nor did it make an appreciable difference in respondent self-reported mood.

The comments used in the sample group are perhaps worse than many Internet comments. They also represent only a small sample of the whopping 7,725 comments—many of them negative or downright offensive—on the actual article. It's easy to see how a reader might reassess her opinion of an article after catching a glimpse of thousands of negative opinions about it.

There are good options for encouraging reader feedback: nice moderated comment sections, forums that build community, quick exchanges on Twitter, or lengthy feedback over email. But unmoderated comments appear to have a small, but real deleterious effect on readers' perception of the sites on which they appear. And that appears to have implications for the bottom-line metric on the web: traffic.