Evelyn Douek: The internet’s titans make a power grab
The tweet that had landed me in social-media jail was one I’d sent the night before. I have been co-hosting a daily live-streamed show with Atlantic contributing writer Benjamin Wittes since the beginning of the pandemic, and our guest last Tuesday night had been the writer Molly Jong-Fast. In the midst of the show, she had had a sotto voce conversation with her spouse, who had reached on camera to try to take a plate away. Jong-Fast had jokingly responded, “If you take that I will kill you,” before turning back to the camera and smilingly saying, “Working at home is a delightful adventure.” Although her response was funny in the moment, my quotation of it in a tweet didn’t exactly capture the humor.
But I wasn’t suspended from Twitter for telling a bad joke. Because I’d included the words I will kill you, I was banned for violating Twitter’s rules about incitement to violence. In the simple notification I received from the company, I could see a lot that many users probably couldn’t. Content that violates the rules of a social-media platform gets flagged in two ways: by other users reporting it or through algorithmic identification. After the recent shootings in Kenosha, Wisconsin, social-media companies came under intense scrutiny to take down posts and accounts that threatened or called for violence. In such a moment, Twitter can cast a wide net for potential infractions by using algorithms that also generate a large number of false positives—say, by flagging lots of postings that include the word kill.
Once a posting is flagged as potentially problematic speech, a human likely still has to review it—but that human is probably a contract worker at a call center–type facility outside the United States. That’s not necessarily a problem, but what counts as humor, satire, or political protest depends heavily on cultural context. In addition, many such centers have been shut down because of the pandemic, and the number of humans reviewing content has been dramatically reduced—making accurate content moderation even harder.
Graeme Wood: Kyle Rittenhouse, Kenosha, and the sheepdog mentality
So an overly literal and aggressive AI likely flagged my speech, a human likely reviewed it and did not get the humor—understandable—and removed it. In normal times this would have meant just my tweet coming down, but these are not normal times. So my entire account was suspended. Right now, the public-relations cost for Twitter of keeping up an account that might be spreading violent propaganda is very high—and the downside to the company of suspending the average user is very low.
But I am in a better position than most users to potentially make trouble for the company. Not only do I study speech rules for a living, but I also talk to a lot of journalists about content moderation. Because of my professional focus, I also have an unusually high level of access to people at the company. Which is presumably why, less than an hour after suspending my account, the company reinstated it.