Facebook Doesn't Understand Itself

The company’s content moderation guide suggests it hasn’t come to grips with its unique role in the world.

Robert Galbraith / Reuters

Facebook’s 2 billion users post a steady stream of baby pictures, opinions about romantic comedies, reactions to the news—and disturbing depictions of violence, abuse, and self-harm. Over the last decade, the company has struggled to come to terms with moderating that last category. How do they parse a joke from a threat, art from pornography, a cry for help from a serious suicide attempt? And even if they can correctly categorize disturbing posts with thousands of human contractors sifting through user-flagged content, what should they do about it?

This weekend, The Guardian began publishing stories based on 100 documents leaked to them from the training process that these content moderators go through. They’re calling it The Facebook Files. Facebook neither confirmed nor denied the authenticity of the documents, but given The Guardian’s history of reporting from leaks, we proceed here with the assumption that the documents are real training materials used by at least one of Facebook’s content moderation contractors.

The Guardian has so far focused on specific types of cases that come up in content moderation: the abuse of children and animals, revenge porn, self-harm, and threats of violence.

The moderator training guidelines are filled with examples. Some show moderators being trained to allow remarkably violent statements to stay on the site. This one, for example, is supposed to help content moderators see the difference between “credible” threats of violence and other statements invoking violence.

Check marks mean the statements can stay on Facebook. X marks mean the statements should be deleted by moderators (The Guardian).

The slides suggest that Facebook has begun to come up with rules that cover literally anything distressing or horrible someone could post. But what do they say about the role Facebook sees itself playing in the world it's creating?

In explaining the company’s reasoning about violent posts, a training document says, “We aim to allow as much speech as possible but draw the line at content that could credibly cause real-world harm.”

In the U.S., there is obviously an entire body of legal cases dedicated to parsing the limits and protections of speech. Different places in the world have different rules and norms. But these cases occur in the context of a single national government and its relationship to “free speech.”

Here, we’re talking about a platform, not a government. Facebook is unconstrained by centuries of interpretations of constitutions and legal precedents. It could do whatever it wanted.

They could systematically aim for harm minimization not speech maximization. That change of assumptions would lead to a different set of individual guidelines on posts. The popular children's online world, Club Penguin, for example, offered multiple levels of language filtering as well as an "Ultimate Safe Chat" mode that only allowed pre-selected phrases to be chosen from a list. At one point, a thousand words were being added to the software's verboten list per day. But “allow[ing] as much speech as possible” has been part of the ideology of this generation of social media companies from the very beginning.

Getting people to post more, as opposed to less, is the core of Facebook’s mission as a company. It is no surprise that the companies built on sharing that have been the most successful come from the United States, which is the most pro-free speech country in the world.

From these documents and the company’s statements, the company has pragmatically chosen to limit areas where it has encountered problems. And those problems are primarily quantified through the flagging that users themselves do.

“As a trusted community of friends, family, coworkers, and classmates, Facebook is largely self-regulated,” one document reads. “People who use Facebook can and do report content that they find questionable or offensive.”

Facebook wants to stay out of it. So Facebook reacts, evolving content moderation guidelines to patch the holes where “self-regulation” fails. Given the number of territories and cultures into which Facebook has integrated itself, one can imagine Facebook’s leadership sees this both as the most reasonable and only practical approach. In cases where they have deployed top-down speech limits, they’ve gotten it wrong, too (as in the “Napalm Girl” controversy).

“We work hard to make Facebook as safe as possible while enabling free speech,” said Monika Bickert, Facebook’s Head of Global Policy Management. “This requires a lot of thought into detailed and often difficult questions, and getting it right is something we take very seriously.”

Let’s stipulate that these are difficult decisions on an individual basis. And let’s further stipulate that multiplying the problem by 2-billion users makes the task daunting, even for a company with $7 billion on hand. Facebook has committed to adding 3,000 more content moderators to the 4,500 working for the company today.

But is Facebook’s current approach to content moderation built on a firm foundation? The company’s approach to content moderation risks abdicating the responsibility that the world’s most popular platform needs to take on.

“When millions of people get together to share things that are important to them, sometimes these discussions and posts include controversial topics and content,” we read in the training document.  “We believe this online dialogue mirrors the exchange of ideas and opinions that happens throughout people’s lives offline, in conversations at home, at work, in cafes and in classrooms.”

In other words, Facebook holds that the posts on its platform reflect offline realities and are merely a reflection of what is, rather than a causal factor in making things come to be.

Facebook must accept the reality that it has changed how people talk to each other. When we have conversations “at home, at work, in cafes, and in classrooms,” there is not an elaborate scoring methodology that determines whose voice will be the loudest. Russian trolls aren’t interjecting disinformation. My visibility to my family is not dependent on the quantifiable engagement that my statements generate. Every word that I utter or picture that I like is not being used to target advertisements (including many from media companies and political actors) at me.

The platform’s own dynamics are a huge part of what gets posted to the platform. They are less a “mirror” of social dynamics than an engine driving them to greater intensity, with unpredictable consequences.

Facebook’s Mark Zuckerberg seemed to acknowledge this in his epic manifesto about the kind of community that he wanted Facebook to build.

“For the past decade, Facebook has focused on connecting friends and families,” he wrote. “With that foundation, our next focus will be developing the social infrastructure for community—for supporting us, for keeping us safe, for informing us, for civic engagement, and for inclusion of all.”

To get this “social infrastructure for community” right, Facebook has to acknowledge that it has not merely “connected friends and families." It has changed their very nature.