Flawed Analysis of Drone Strike Data Is Misleading Americans

More

The media's go to source for kill figures is the New America Foundation. But its invaluable work is being cited in support of conclusions it doesn't support.

drone sky full reuters .jpg
Reuters

Peter Bergen is among the most influential people in America when it comes to shaping public attitudes about drone strikes inside Pakistan. An author, print journalist, and broadcaster, he is a national security analyst at CNN, a fellow at Fordham University's Center on National Security, and the director of the New America Foundation's National Security Studies Program. It's that last position that is most important for our purposes, for the New America Foundation sponsors "The Year of the Drone," an invaluable effort to analyze press reports on drone strikes. "Our research draws only on accounts from reliable media organizations with deep reporting capabilities in Pakistan," Bergen wrote, "and reports in the leading English-language newspapers in Pakistan--the Daily Times, Dawn, and the News--as well as those from Geo TV."

Using reports of drone strikes in those outlets, the New America Foundation does its best to determine the date an attack occurs, the number of "militants" killed, and the number of civilian deaths. Everyone interested in those subjects is indebted to the organization for the work it has done aggregating disparate information. But Bergen and others are repeatedly overstating the conclusions that can be draw from their research. As a result of this wrongheaded analysis, published most prominently at CNN, countless Americans are being misled about our drone war.

I noticed this problem earlier this month, when a graphic published at the top of a Bergen column at CNN asserted that zero innocents have been killed during drone strikes carried out in Pakistan this year. As I explained at length, there is no way to confirm that conclusion from the New America Foundation's data. The Bureau of Investigative Journalism, a British run enterprise that tracks drone strikes, has since weighed in agreeing with my analysis, concluding in part that "Bergen's claim of zero reported civilian casualties this year is... factually inaccurate."

On July 14, Bergen published another problematic article at CNN, "Civilian Casualties Plummet in Drone Strikes." I'll address its dubious assertions first and note its troubling omissions afterward.

Here's the core of Bergen's piece:

The New America Foundation has been collecting data about the drone attacks systematically for the past three years from reputable news sources such as the New York Times and Reuters, as well as Pakistani media outlets such as the Express Tribune and Dawn. According to the data generated by averaging the high and low casualty estimates of militant and civilian deaths published in a wide range of those outlets, the estimated civilian death rate in U.S. drone strikes in Pakistan has declined dramatically since 2008, when it was at its peak of almost 50%.

Today, for the first time, the estimated civilian death rate is at or close to zero. Over the life of the drone program in Pakistan, which began with a relatively small number of strikes between 2004 and 2007, the estimated civilian death rate is 16%. And in the Obama administration, between 1,507 and 2,438 people have been killed in drone strikes. Of those, 148 to 309, or between 10% and 12%, were civilians, according to the New America Foundation data.

Drawing conclusions that specific from the New America Foundation's data is completely unjustified. To understand why, think about the nature of the information they've gathered. Without even reading the specific press accounts, it's obvious that drone strikes unreported in the press are completely invisible. Restricting ourselves to the drone strikes that do make it into the press, how reliable are the numbers given for total killed, militants killed, and civilians killed?

The short answer: not very reliable.

For a longer, more specific critique, I was steered to collaborative research being conducted at two law clinics, New York University's Global Justice Clinic and Stanford's International Human Rights and Conflict Resolution Clinic. Their report on drone strikes in Pakistan is coming out in a couple weeks. Professor Sarah Knuckey of NYU provided me with findings she's pulled together with help from Christopher Holland, a former student in the Global Justice Clinic. They analyzed the stories cited by the New America Foundation as the basis for its report on drone strikes in 2012.

What follows is my summary of their findings:

  • The New America Foundation cited 96 articles as the basis for their conclusion that there have been zero civilian casualties in Pakistan during 2012. Links to ten of the articles didn't work, so the NYU team ultimately analyzed 86 total articles from a total of 13 different news sources.
  • In 74 percent of the articles, the only source for the number of "militants" killed was anonymous government officials (almost always unnamed Pakistani officials).
  • The New America Foundation documented 27 separate drone strikes in 2012. In more than half of those drone strikes -- 16 of them, to be exact -- all information about the number of "militants" killed came from unnamed government officials (almost always unnamed Pakistani officials).
  • In 15 articles cited by the New America Foundation sources said the identities of those killed could not be identified. And in 18 articles, the "compound" or other object of attack was said to be "destroyed," calling into question estimates about numbers of casualties and the identities of the dead.

In his work on drones, Bergen pointedly notes that the New America Foundation relies on reports "from reliable media organizations with deep reporting capabilities in Pakistan," as though "deep reporting" is a prerequisite if New America is to treat a casualty estimate as reliable. But many of the accounts that form the basis of New America estimates are no more deeply reported than getting an unnamed official to state the number of deaths, which is taken on faith because it's the only information available. There is, of course, reason to doubt the accuracy of both Pakistani intelligence and government officials, especially those who are only willing to speak off the record.

Yet Bergen writes in his CNN piece that the New America Foundation's estimate of innocents killed is "far below the civilian death rate that the Pakistani government and other private research groups such as Pakistan Body Count have claimed. A report released by Pakistani authorities in 2010 estimated that for every militant killed in a drone strike in 2009, 140 Pakistani civilians also died, and that the civilian casualty rate for that year was more than 90%." In other words, confronted with an official estimate of civilians killed from Pakistan's government, Bergen treats it skeptically, and concludes that it is wildly wrong. Yet when an anonymous Pakistani officials gives an estimate that is filtered through any one of 13 press outlets, Bergen treats it as reliable, using it to inform his analysis of the actual number of innocents killed.

Jump to comments
Presented by

Conor Friedersdorf is a staff writer at The Atlantic, where he focuses on politics and national affairs. He lives in Venice, California, and is the founding editor of The Best of Journalism, a newsletter devoted to exceptional nonfiction.

Get Today's Top Stories in Your Inbox (preview)

A Fascinating Short Film About the Multiverse

If life is a series of infinite possibilities, what does it mean to be alive?


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

The Death of Film

You'll never hear the whirring sound of a projector again.

Video

How to Hunt With Poison Darts

A Borneo hunter explains one of his tribe's oldest customs: the art of the blowpipe

Video

A Delightful, Pixar-Inspired Cartoon

An action figure and his reluctant sidekick trek across a kitchen in search of treasure.

Video

I Am an Undocumented Immigrant

"I look like a typical young American."

Video

Why Did I Study Physics?

Using hand-drawn cartoons to explain an academic passion

Writers

Up
Down

More in Politics

Just In