This is huge progress, she says. By inviting experts who understand the roots of violence against women and children and are familiar with emerging strategies to prevent it, tech is more likely to innovate improvements. The once profusely applied “Controversial Humor” label in Facebook is no longer in use. The company now officially recognizes gender-based hate as a legitimate concern, and its representatives continue to work closely with advocates like Southworth and the coalition that formed during the #FBRape campaign. There are ongoing efforts to improve user safety and identify content that is threatening, harassing, hateful, or discriminatory.
Southworth calls the company’s representatives “thoughtful, passionate, concerned, and straddling the line between free speech and safety.” But, sometimes, progress feels slow. “The teams who handle these cases are just swamped,” she explained.
When Emily Bazelon, author of a book and a March 2013 Atlantic story about Internet bullies, visited Facebook’s headquarters, the young men she saw working as moderators were spending roughly 30 seconds assessing each reported post, millions of reports a week. Outsourced speech moderation has become a booming industry. Like Facebook’s own moderation process, the operations of these companies are opaque by design.
TaskUs, with bases in Santa Monica, California, and several locations in the Philippines, provides moderation for iPhone and Android apps such as Whisper, Secret, and Yik Yak. The company advertises “a bulletproof system to ensure that no one—not a single person—is hurt physically or mentally by the actions of another user in an anonymous app community.” Yet TaskUs doesn’t disclose its standards of speech, its hiring practices, its training process, or working conditions. “Unfortunately,” we were told when we inquired, “we’re bound by confidentiality from discussing details of process including hiring and training. We can speak generally about how we handle the moderation process but our clients are not comfortable with us exposing anything proprietary (and they consider the moderation and training processes proprietary).”
While private companies protect their practices, nonprofits like The Internet Watch Foundation don’t. IWF, based in Cambridge, England, screens images of child sexual abuse for Facebook, Google, and Virgin Media, among others. IWF staff watch, analyze, categorize, and report abusive images—70 percent of them involve children under 10. Data collected by police across England and Wales in 2012 suggest that 150 million child pornography images were in distribution in the UK alone that year. By comparison, in 1995, when the reach of the Internet was far narrower, only about 7,000 child pornography images were in online circulation.
IWF’s analysts see everything, said Heidi Kempster, IWF’s director of business affairs, during a conversation this summer. Kempster was candid about IWF’s business practices: The group screens rigorously during its hiring processes, conducting psychological interviews that establish everything from family history and relationship-building skills to views on pornography. The company also requires monthly individual counseling and quarterly group counseling, as well as expert consultation—with police, attorneys, or judges— and breaks as needed. IWF analysts, said Kempster, “look at shocking and violent images all day every day. There are days that are tough. They have to take time out.”