NewsTrust on Fact-Checking and the Wisdom of Crowds

What role do journalists, nonprofits, and other partners play in your process? It seems they have the final word as "Judges." Is the process hands off up until that point? Are the opinions of your judges ever in tension with significant portions of your community?

We decided early on that professionals needed to guide this collaborative investigation process. We wanted to avoid some of the pitfalls of pure crowdsourcing initiatives, which can turn into mob scenes -- particularly around politically-charged issues. This is why we are partnering with the Center for Public Integrity, whose newsroom of investigative reporters will play a crucial role in this project.

These journalists perform a variety of tasks throughout each investigation: select which claims to feature for fact-checking, search for (and link to) factual evidence about these claims, guide citizen researchers, monitor their answers, moderate their comments -- and write a final verdict based on our collective findings. Community members can post as many links and comments as they want, and can change their answers at any time. These community answers are tallied on our claim pages, but our professional editors have the final say in deciding our verdict.

Our community's responses are generally consistent with verdicts from our Truthsquad editors. However, there have been some cases where findings from citizens and editors have diverged. In cases like these, we go out of our way to carefully examine dissenting views, often citing them in our final reports, with references to the facts that led us to reach a different conclusion. We also invite the community to challenge our final verdicts after publication, and have on occasion changed our verdict when new factual evidence was provided to warrant an update.

Overall, we are pleased that comments from participants are typically civil, and that they seem genuinely engaged in this communal quest for credible information.

Truthsquad has existed as a pilot since 2010. What lessons have you learned? How has your fact-checking process evolved since then?

NewsTrust created a first pilot for Truthsquad in July 2010, with funding from Omidyar Network and the MacArthur Foundation. At the start of this experiment, we asked experienced journalists at and the Poynter Institute to coach us and our community and help write and edit some of our first verdicts. This winter, we conducted a second pilot with and, focusing on statements from reporters or commentators -- and we also hosted a local Truthsquad in Maryland, as part of our NewsTrust Baltimore pilot.

Here are our key takeaways from our Truthsquad pilots so far:

  • A game-like experience makes fact-checking more engaging.
  • A "pro-am" collaboration delivers more reliable results.
  • Crowd contributions are limited, requiring oversight and rewards.
  • Truthsquad fills a gap between journalism and social media.

But the overall process of collaboration between experienced journalists and citizens will remain in place, as we found it particularly effective for informing and engaging each other in this interactive quest.

One question that all fact checkers have to grapple with is what counts as fact. When I interviewed Brooks Jackson of, I asked him about how they decided to weigh in on the impacts of the stimulus bill, a tremendously complicated issue. How do you decide what sort of facts to check? Are some issues too complicated to take on?

We generally look for controversial claims about important public issues, and try to fact-check influential newsmakers from across the political spectrum. We focus on "statements of fact" that can be verified with the right amount of research, ideally with clear-cut answers -- so the community does not get bogged down in tasks that cannot be completed effectively.

That said, we have on occasion taken on some difficult challenges, such as this claim from Sen. Richard Durbin, who stated that "Social Security does not add one penny to the deficit." The wide difference between legal and economic interpretations of this politically-charged issue was a major source of disagreement for our community -- and it has polarized Americans across party lines, with Democrats claiming that Social Security has no impact on the deficit, and Republicans claiming the opposite.

Our editors were divided on this issue as well and took turns arguing for different perspectives before agreeing to a final verdict. After several days of extensive research, we concluded that Durbin's statement did not account for the economic interpretation of Social Security's actual costs on a cash basis, and dismissed the significant impact of the "payroll tax holiday." For these reasons, we found it "half true." This was also the first time that the verdict from our editors had diverged so widely from other NewsTrust members (over two-thirds of participants thought Senator Durbin's claim was true). We attribute this discrepancy to the charged political climate, the large number of liberals on our site, as well as the complexity of the issue, which is subject to interpretation and requires a long time to research thoroughly.

But despite the complexity of this task, we found this particular investigation to be a very rewarding experience for our team and community. We collectively learned a lot more about Social Security and its impact on the U.S. budget, as well as how different interpretations can lead to diverging conclusions.

Perhaps the most important insight from this experiment is that reality is nuanced and comes in many shades of gray, despite our instinctive desire for black-and-white answers that match our beliefs. At the same time, we have also discovered that when people of goodwill come together with a shared focus on facts and some professional guidance, they can shed light on the most complex issues and learn a great deal from each other.

NewsTrust offers an interesting set of tools, like widgets and bookmarklets, to integrate its work into the rest of the Web. How do you see Truthsquad fitting into the rest of the news ecosystem?

Besides the site, we plan to offer the Truthsquad service to our customers wherever they are, whether on partner sites, on social networks, or on mobile platforms. For example, we are developing Truthsquad widgets and bookmarklets based on our current NewsTrust tools -- and we also aim to provide a variety of APIs and mobile applications as our service expands.

In today's rapidly changing news ecosystem, content is becoming increasingly fragmented and delivered over a wide range of platforms. As a result, we are shifting our strategy away from a single consumer destination, aiming instead to serve our customers on the platforms of their choice -- with a special focus on our partner sites.

For now, I encourage your readers to sign up on our pilot site, so we can inform them when our new service launches in coming months.

Image: NewsTrust.


Presented by

Walter Frick is an associate editor at Harvard Business Review.

The Horrors of Rat Hole Mining

"The river was our source of water. Now, the people won't touch it. They are repulsed by it."

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register with Disqus.

Please note that The Atlantic's account system is separate from our commenting system. To log in or register with The Atlantic, use the Sign In button at the top of every page.

blog comments powered by Disqus


The Horrors of Rat Hole Mining

"The river was our source of water. Now, the people won't touch it."


What's Your Favorite Slang Word?

From "swag" to "on fleek," tweens choose.


Cryotherapy's Dubious Appeal

James Hamblin tries a questionable medical treatment.


Confessions of Moms Around the World

In Europe, mothers get maternity leave, discounted daycare, and flexible working hours.


How Do Trees Know When It's Spring?

The science behind beautiful seasonal blooming

More in Technology

Just In