Flawed Analysis of Drone Strike Data Is Misleading Americans

The media's go to source for kill figures is the New America Foundation. But its invaluable work is being cited in support of conclusions it doesn't support.

drone sky full reuters .jpg
Reuters

Peter Bergen is among the most influential people in America when it comes to shaping public attitudes about drone strikes inside Pakistan. An author, print journalist, and broadcaster, he is a national security analyst at CNN, a fellow at Fordham University's Center on National Security, and the director of the New America Foundation's National Security Studies Program. It's that last position that is most important for our purposes, for the New America Foundation sponsors "The Year of the Drone," an invaluable effort to analyze press reports on drone strikes. "Our research draws only on accounts from reliable media organizations with deep reporting capabilities in Pakistan," Bergen wrote, "and reports in the leading English-language newspapers in Pakistan--the Daily Times, Dawn, and the News--as well as those from Geo TV."

Using reports of drone strikes in those outlets, the New America Foundation does its best to determine the date an attack occurs, the number of "militants" killed, and the number of civilian deaths. Everyone interested in those subjects is indebted to the organization for the work it has done aggregating disparate information. But Bergen and others are repeatedly overstating the conclusions that can be draw from their research. As a result of this wrongheaded analysis, published most prominently at CNN, countless Americans are being misled about our drone war.

I noticed this problem earlier this month, when a graphic published at the top of a Bergen column at CNN asserted that zero innocents have been killed during drone strikes carried out in Pakistan this year. As I explained at length, there is no way to confirm that conclusion from the New America Foundation's data. The Bureau of Investigative Journalism, a British run enterprise that tracks drone strikes, has since weighed in agreeing with my analysis, concluding in part that "Bergen's claim of zero reported civilian casualties this year is... factually inaccurate."

On July 14, Bergen published another problematic article at CNN, "Civilian Casualties Plummet in Drone Strikes." I'll address its dubious assertions first and note its troubling omissions afterward.

Here's the core of Bergen's piece:

The New America Foundation has been collecting data about the drone attacks systematically for the past three years from reputable news sources such as the New York Times and Reuters, as well as Pakistani media outlets such as the Express Tribune and Dawn. According to the data generated by averaging the high and low casualty estimates of militant and civilian deaths published in a wide range of those outlets, the estimated civilian death rate in U.S. drone strikes in Pakistan has declined dramatically since 2008, when it was at its peak of almost 50%.

Today, for the first time, the estimated civilian death rate is at or close to zero. Over the life of the drone program in Pakistan, which began with a relatively small number of strikes between 2004 and 2007, the estimated civilian death rate is 16%. And in the Obama administration, between 1,507 and 2,438 people have been killed in drone strikes. Of those, 148 to 309, or between 10% and 12%, were civilians, according to the New America Foundation data.

Drawing conclusions that specific from the New America Foundation's data is completely unjustified. To understand why, think about the nature of the information they've gathered. Without even reading the specific press accounts, it's obvious that drone strikes unreported in the press are completely invisible. Restricting ourselves to the drone strikes that do make it into the press, how reliable are the numbers given for total killed, militants killed, and civilians killed?

The short answer: not very reliable.

For a longer, more specific critique, I was steered to collaborative research being conducted at two law clinics, New York University's Global Justice Clinic and Stanford's International Human Rights and Conflict Resolution Clinic. Their report on drone strikes in Pakistan is coming out in a couple weeks. Professor Sarah Knuckey of NYU provided me with findings she's pulled together with help from Christopher Holland, a former student in the Global Justice Clinic. They analyzed the stories cited by the New America Foundation as the basis for its report on drone strikes in 2012.

What follows is my summary of their findings:

  • The New America Foundation cited 96 articles as the basis for their conclusion that there have been zero civilian casualties in Pakistan during 2012. Links to ten of the articles didn't work, so the NYU team ultimately analyzed 86 total articles from a total of 13 different news sources.
  • In 74 percent of the articles, the only source for the number of "militants" killed was anonymous government officials (almost always unnamed Pakistani officials).
  • The New America Foundation documented 27 separate drone strikes in 2012. In more than half of those drone strikes -- 16 of them, to be exact -- all information about the number of "militants" killed came from unnamed government officials (almost always unnamed Pakistani officials).
  • In 15 articles cited by the New America Foundation sources said the identities of those killed could not be identified. And in 18 articles, the "compound" or other object of attack was said to be "destroyed," calling into question estimates about numbers of casualties and the identities of the dead.

In his work on drones, Bergen pointedly notes that the New America Foundation relies on reports "from reliable media organizations with deep reporting capabilities in Pakistan," as though "deep reporting" is a prerequisite if New America is to treat a casualty estimate as reliable. But many of the accounts that form the basis of New America estimates are no more deeply reported than getting an unnamed official to state the number of deaths, which is taken on faith because it's the only information available. There is, of course, reason to doubt the accuracy of both Pakistani intelligence and government officials, especially those who are only willing to speak off the record.

Presented by

Conor Friedersdorf is a staff writer at The Atlantic, where he focuses on politics and national affairs. He lives in Venice, California, and is the founding editor of The Best of Journalism, a newsletter devoted to exceptional nonfiction.

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well. Bestselling author Mark Bittman teaches James Hamblin the recipe that everyone is Googling.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well.

Video

Before Tinder, a Tree

Looking for your soulmate? Write a letter to the "Bridegroom's Oak" in Germany.

Video

The Health Benefits of Going Outside

People spend too much time indoors. One solution: ecotherapy.

Video

Where High Tech Meets the 1950s

Why did Green Bank, West Virginia, ban wireless signals? For science.

Video

Yes, Quidditch Is Real

How J.K. Rowling's magical sport spread from Hogwarts to college campuses

Video

Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

More in Politics

Just In