NewsTrust on Fact-Checking and the Wisdom of Crowds
NewsTrust is shifting its focus to devote more time and resources to TruthSquad.com, a collaborative fact-checking site it launched in 2010
Last week in New Hampshire, Texas Governor and Republican presidential candidate Rick Perry challenged the validity of climate science, saying, "I think we're seeing it almost weekly or even daily, scientists who are coming forward and questioning the original idea that man-made global warming is what is causing the climate to change."
Luckily, the Pulitzer Prize-winning site Politifact.com came to the rescue by fact-checking the governor's claim, which it found to be false. The Washington Post's fact checker looked at the same statement and declared that "the governor is willfully ignoring the facts and making false accusations based on little evidence." But fact-checking politicians heading into election season is akin to playing Whack-A-Mole. No matter how many claims these and other fact checkers beat back, new whoppers keep popping up.
But what if you could scale these laudable efforts by harnessing the wisdom of the crowds? That's what NewsTrust, a nonprofit news aggregator, set out to do when it piloted its collaborative fact-checking site Truthsquad.com in 2010. Now, just over a year later, NewsTrust is shifting its focus to devote more time and resources to that effort.
I spoke with NewsTrust executive director Fabrice Florin by email to learn more about how he and his team plan to leverage crowdsourcing to fact-check the 2012 elections. For more on traditional fact-checking, see my interview with Factcheck.org's Brooks Jackson.
NewsTrust is shifting its emphasis, from filtering and aggregating quality journalism from other sources to crowdsourced fact-checking. Tell me more about NewsTrust, about what you have been doing, and why you decided to shift your focus.
NewsTrust helps people find good journalism online, so they can make more informed decisions as citizens. We're nonprofit, nonpartisan, and committed to factual information, news literacy and civic engagement. NewsTrust.net, our social news network, features a daily feed of quality news and opinions from mainstream and independent sources, based on ratings from our reviewers.
Our Web review tools enable the public to evaluate accuracy, fairness, context, and other core journalistic principles -- and help people become more discerning news consumers in the process. Since we launched our site in 2006, we have attracted a growing community of citizens, students, educators, and journalists, who share a passion for quality news and information, serving over 1.3 million visitors last year. Our media partners include the Washington Post, USA Today, and PBS NewsHour -- and our educational partners include Stanford University, Stony Brook, and Towson University, to name but a few.
We're proud of what we've accomplished over the last five years, but news sharing on the Web is now shifting to large social networks such as Facebook and Twitter, reducing the need for curated news sites like NewsTrust.net. Instead, we see an emerging need for quality fact-checking services and collaborative evaluation tools, which we think we can effectively provide by extending our innovative social media platform to serve partner communities on their sites.
As a result, NewsTrust is now pivoting from a standalone news curation site to a consultancy that will serve the needs of larger partners and help their communities become better informed about important public issues. Our initial focus will be on fact-checking services to expose misinformation in the public debate. We will also explore partnerships that enable us to provide news literacy and civic engagement services through popular consumer and educational channels.
With your latest project, Truthsquad, you are leveraging crowdsourcing to fact-check political claims. Can you explain to readers what that means in a nutshell?
Truthsquad is a new pro-am fact-checking service we are developing with the Center for Public Integrity, to help journalists and citizens separate fact from fiction together.
We are combining the best practices of crowdsourcing with the knowledge of a large nonprofit newsroom and the reach of major news partners. This unique approach empowers citizens to collaborate with journalists to investigate controversial statements from politicians and newsmakers. Participants are invited to post questionable claims online, research factual evidence supporting or opposing these claims, and verify their accuracy as a community, with professional oversight.
Our goal is to launch a daily service in early 2012 on Truthsquad.com and a few select partner sites. We aim to provide a one-stop destination for fact-checked information -- featuring our own findings, as well as promoting the work of other trusted research organizations like FactCheck.org and PolitiFact. Our advisors include Brooks Jackson (FactCheck.org), Craig Newmark (Craigslist), Dan Gillmor, Jeff Jarvis, Ellen Miller, and Jay Rosen, with more advisors and partners to be announced in coming weeks.
One thing I think you've already gotten right is asking readers not just what they think but what they've seen or read. Users can rate statements true or false and add comments, but they are also encouraged to add links and describe whether the link supports or opposes the claim. What made you decide to add this feature?
For each claim we fact-check (example), our editors provide links to factual evidence supporting or opposing that claim. We then invite our community to review these links before giving a final answer. Participants can also add more links, as described in our pilot FAQ.
The purpose of this key feature is to encourage participants to base their answers on facts, not just their own opinions. Reviewing a variety of links about a claim can help you weigh the evidence from all angles and reach a more informed decision. We also think that getting citizens to reach beyond their comfort zone and regular news sources can help them develop a more open mind, build their research skills and appreciate the value of looking at the world from different perspectives.
You can read more about our approach and overall mission in this article from the Nieman Journalism Lab.
What role do journalists, nonprofits, and other partners play in your process? It seems they have the final word as "Judges." Is the process hands off up until that point? Are the opinions of your judges ever in tension with significant portions of your community?
We decided early on that professionals needed to guide this collaborative investigation process. We wanted to avoid some of the pitfalls of pure crowdsourcing initiatives, which can turn into mob scenes -- particularly around politically-charged issues. This is why we are partnering with the Center for Public Integrity, whose newsroom of investigative reporters will play a crucial role in this project.
These journalists perform a variety of tasks throughout each investigation: select which claims to feature for fact-checking, search for (and link to) factual evidence about these claims, guide citizen researchers, monitor their answers, moderate their comments -- and write a final verdict based on our collective findings. Community members can post as many links and comments as they want, and can change their answers at any time. These community answers are tallied on our claim pages, but our professional editors have the final say in deciding our verdict.
Our community's responses are generally consistent with verdicts from our Truthsquad editors. However, there have been some cases where findings from citizens and editors have diverged. In cases like these, we go out of our way to carefully examine dissenting views, often citing them in our final reports, with references to the facts that led us to reach a different conclusion. We also invite the community to challenge our final verdicts after publication, and have on occasion changed our verdict when new factual evidence was provided to warrant an update.
Overall, we are pleased that comments from participants are typically civil, and that they seem genuinely engaged in this communal quest for credible information.
Truthsquad has existed as a pilot since 2010. What lessons have you learned? How has your fact-checking process evolved since then?
NewsTrust created a first pilot for Truthsquad in July 2010, with funding from Omidyar Network and the MacArthur Foundation. At the start of this experiment, we asked experienced journalists at FactCheck.org and the Poynter Institute to coach us and our community and help write and edit some of our first verdicts. This winter, we conducted a second pilot with MediaBugs.org and RegretTheError.com, focusing on statements from reporters or commentators -- and we also hosted a local Truthsquad in Maryland, as part of our NewsTrust Baltimore pilot.
Here are our key takeaways from our Truthsquad pilots so far:
- A game-like experience makes fact-checking more engaging.
- A "pro-am" collaboration delivers more reliable results.
- Crowd contributions are limited, requiring oversight and rewards.
- Truthsquad fills a gap between journalism and social media.
But the overall process of collaboration between experienced journalists and citizens will remain in place, as we found it particularly effective for informing and engaging each other in this interactive quest.
One question that all fact checkers have to grapple with is what counts as fact. When I interviewed Brooks Jackson of Factcheck.org, I asked him about how they decided to weigh in on the impacts of the stimulus bill, a tremendously complicated issue. How do you decide what sort of facts to check? Are some issues too complicated to take on?
We generally look for controversial claims about important public issues, and try to fact-check influential newsmakers from across the political spectrum. We focus on "statements of fact" that can be verified with the right amount of research, ideally with clear-cut answers -- so the community does not get bogged down in tasks that cannot be completed effectively.
That said, we have on occasion taken on some difficult challenges, such as this claim from Sen. Richard Durbin, who stated that "Social Security does not add one penny to the deficit." The wide difference between legal and economic interpretations of this politically-charged issue was a major source of disagreement for our community -- and it has polarized Americans across party lines, with Democrats claiming that Social Security has no impact on the deficit, and Republicans claiming the opposite.
Our editors were divided on this issue as well and took turns arguing for different perspectives before agreeing to a final verdict. After several days of extensive research, we concluded that Durbin's statement did not account for the economic interpretation of Social Security's actual costs on a cash basis, and dismissed the significant impact of the "payroll tax holiday." For these reasons, we found it "half true." This was also the first time that the verdict from our editors had diverged so widely from other NewsTrust members (over two-thirds of participants thought Senator Durbin's claim was true). We attribute this discrepancy to the charged political climate, the large number of liberals on our site, as well as the complexity of the issue, which is subject to interpretation and requires a long time to research thoroughly.
But despite the complexity of this task, we found this particular investigation to be a very rewarding experience for our team and community. We collectively learned a lot more about Social Security and its impact on the U.S. budget, as well as how different interpretations can lead to diverging conclusions.
Perhaps the most important insight from this experiment is that reality is nuanced and comes in many shades of gray, despite our instinctive desire for black-and-white answers that match our beliefs. At the same time, we have also discovered that when people of goodwill come together with a shared focus on facts and some professional guidance, they can shed light on the most complex issues and learn a great deal from each other.
NewsTrust offers an interesting set of tools, like widgets and bookmarklets, to integrate its work into the rest of the Web. How do you see Truthsquad fitting into the rest of the news ecosystem?
Besides the Truthsquad.com site, we plan to offer the Truthsquad service to our customers wherever they are, whether on partner sites, on social networks, or on mobile platforms. For example, we are developing Truthsquad widgets and bookmarklets based on our current NewsTrust tools -- and we also aim to provide a variety of APIs and mobile applications as our service expands.
In today's rapidly changing news ecosystem, content is becoming increasingly fragmented and delivered over a wide range of platforms. As a result, we are shifting our strategy away from a single consumer destination, aiming instead to serve our customers on the platforms of their choice -- with a special focus on our partner sites.
For now, I encourage your readers to sign up on our Truthsquad.com pilot site, so we can inform them when our new service launches in coming months.
Image: NewsTrust.
.this>