Flawed Analysis of Drone Strike Data Is Misleading Americans

The media's go to source for kill figures is the New America Foundation. But its invaluable work is being cited in support of conclusions it doesn't support.

drone sky full reuters .jpg

Peter Bergen is among the most influential people in America when it comes to shaping public attitudes about drone strikes inside Pakistan. An author, print journalist, and broadcaster, he is a national security analyst at CNN, a fellow at Fordham University's Center on National Security, and the director of the New America Foundation's National Security Studies Program. It's that last position that is most important for our purposes, for the New America Foundation sponsors "The Year of the Drone," an invaluable effort to analyze press reports on drone strikes. "Our research draws only on accounts from reliable media organizations with deep reporting capabilities in Pakistan," Bergen wrote, "and reports in the leading English-language newspapers in Pakistan--the Daily Times, Dawn, and the News--as well as those from Geo TV."

Using reports of drone strikes in those outlets, the New America Foundation does its best to determine the date an attack occurs, the number of "militants" killed, and the number of civilian deaths. Everyone interested in those subjects is indebted to the organization for the work it has done aggregating disparate information. But Bergen and others are repeatedly overstating the conclusions that can be draw from their research. As a result of this wrongheaded analysis, published most prominently at CNN, countless Americans are being misled about our drone war.

I noticed this problem earlier this month, when a graphic published at the top of a Bergen column at CNN asserted that zero innocents have been killed during drone strikes carried out in Pakistan this year. As I explained at length, there is no way to confirm that conclusion from the New America Foundation's data. The Bureau of Investigative Journalism, a British run enterprise that tracks drone strikes, has since weighed in agreeing with my analysis, concluding in part that "Bergen's claim of zero reported civilian casualties this year is... factually inaccurate."

On July 14, Bergen published another problematic article at CNN, "Civilian Casualties Plummet in Drone Strikes." I'll address its dubious assertions first and note its troubling omissions afterward.

Here's the core of Bergen's piece:

The New America Foundation has been collecting data about the drone attacks systematically for the past three years from reputable news sources such as the New York Times and Reuters, as well as Pakistani media outlets such as the Express Tribune and Dawn. According to the data generated by averaging the high and low casualty estimates of militant and civilian deaths published in a wide range of those outlets, the estimated civilian death rate in U.S. drone strikes in Pakistan has declined dramatically since 2008, when it was at its peak of almost 50%.

Today, for the first time, the estimated civilian death rate is at or close to zero. Over the life of the drone program in Pakistan, which began with a relatively small number of strikes between 2004 and 2007, the estimated civilian death rate is 16%. And in the Obama administration, between 1,507 and 2,438 people have been killed in drone strikes. Of those, 148 to 309, or between 10% and 12%, were civilians, according to the New America Foundation data.

Drawing conclusions that specific from the New America Foundation's data is completely unjustified. To understand why, think about the nature of the information they've gathered. Without even reading the specific press accounts, it's obvious that drone strikes unreported in the press are completely invisible. Restricting ourselves to the drone strikes that do make it into the press, how reliable are the numbers given for total killed, militants killed, and civilians killed?

The short answer: not very reliable.

For a longer, more specific critique, I was steered to collaborative research being conducted at two law clinics, New York University's Global Justice Clinic and Stanford's International Human Rights and Conflict Resolution Clinic. Their report on drone strikes in Pakistan is coming out in a couple weeks. Professor Sarah Knuckey of NYU provided me with findings she's pulled together with help from Christopher Holland, a former student in the Global Justice Clinic. They analyzed the stories cited by the New America Foundation as the basis for its report on drone strikes in 2012.

What follows is my summary of their findings:

  • The New America Foundation cited 96 articles as the basis for their conclusion that there have been zero civilian casualties in Pakistan during 2012. Links to ten of the articles didn't work, so the NYU team ultimately analyzed 86 total articles from a total of 13 different news sources.
  • In 74 percent of the articles, the only source for the number of "militants" killed was anonymous government officials (almost always unnamed Pakistani officials).
  • The New America Foundation documented 27 separate drone strikes in 2012. In more than half of those drone strikes -- 16 of them, to be exact -- all information about the number of "militants" killed came from unnamed government officials (almost always unnamed Pakistani officials).
  • In 15 articles cited by the New America Foundation sources said the identities of those killed could not be identified. And in 18 articles, the "compound" or other object of attack was said to be "destroyed," calling into question estimates about numbers of casualties and the identities of the dead.

In his work on drones, Bergen pointedly notes that the New America Foundation relies on reports "from reliable media organizations with deep reporting capabilities in Pakistan," as though "deep reporting" is a prerequisite if New America is to treat a casualty estimate as reliable. But many of the accounts that form the basis of New America estimates are no more deeply reported than getting an unnamed official to state the number of deaths, which is taken on faith because it's the only information available. There is, of course, reason to doubt the accuracy of both Pakistani intelligence and government officials, especially those who are only willing to speak off the record.

Yet Bergen writes in his CNN piece that the New America Foundation's estimate of innocents killed is "far below the civilian death rate that the Pakistani government and other private research groups such as Pakistan Body Count have claimed. A report released by Pakistani authorities in 2010 estimated that for every militant killed in a drone strike in 2009, 140 Pakistani civilians also died, and that the civilian casualty rate for that year was more than 90%." In other words, confronted with an official estimate of civilians killed from Pakistan's government, Bergen treats it skeptically, and concludes that it is wildly wrong. Yet when an anonymous Pakistani officials gives an estimate that is filtered through any one of 13 press outlets, Bergen treats it as reliable, using it to inform his analysis of the actual number of innocents killed.

That brings us to the information that never appears in Bergen's articles.

Most problematically, he separates drone strike victims into "militants" and "civilians" without ever offering a working definition of "militant." Bergen does note that the Obama Administration reportedly "considers as a 'militant' any military-age male in the strike target area," and assures readers that  "The New America data is not based on the U.S. official definition of a militant and does not rely on any U.S. official counting of the strikes." Eight days ago, I emailed Clara Hogan, a press liaison at New America, asking about its working definition of militant, but received no answer. I emailed Bergen the same question on July 13, but haven't received a reply from him either. As best I can tell, the New America Foundation is following the usage of the various media outlets whose work it draws upon, or the definition used by named and unnamed Pakistani officials, or both, which is problematic, because neither the press nor the Pakistani government seems to have a consistent, rigorous, or transparent definition of what  "militant" means.

It's the weasel word of the drone era.

A precise definition of militant isn't the only omission that Bergen should remedy.

In his latest CNN piece, he writes that "the drop in the number of civilian casualties since 2008 came as a result of several developments, one of which was a directive issued from the White House just days after President Obama took office, to tighten up the way the CIA selected targets and carried out strikes. Specifically, Obama wanted to evaluate and sign off personally on any strike if the agency did not have a 'near certainty' that it would result in zero civilian casualties."

This gives readers the impression that Obama has personally weighed in on every risky strike since 2008, but the fact of the matter is that the CIA doesn't acknowledge the possibility of civilian casualties when all present at the scene of a strike are military-aged males; and the CIA has also launched signature strikes wherein the identities of the human targets are not known to their killers. Surely that detail is relevant. So is this portion from The New York Times story on drones:

...three former senior intelligence officials expressed disbelief that the number could be so low. The C.I.A. accounting has so troubled some administration officials outside the agency that they have brought their concerns to the White House. One called it "guilt by association" that has led to "deceptive" estimates of civilian casualties. "It bothers me when they say there were seven guys, so they must all be militants," the official said. "They count the corpses and they're not really sure who they are."

With signature strikes reportedly down in Pakistan this year, but up in Yemen, where the CIA now has the authority to conduct them, it's also problematic for CNN to title Bergen's piece, "Civilian casualties plummet in drone strikes," when its contents analyze just one of the several countries where we're waging a drone war. It's finally worth noting an exchange between Bergen and the folks at the Bureau of Investigative Journalism.

Says Bergen in his CNN piece:

The London-based Bureau of Investigative Journalism also maintains a count of drone casualties in Pakistan. It reports that between three and 24 civilians have been killed in 2012. The Bureau of Investigative Journalism's high estimate of 24 civilian deaths in 2012 came in part from reports provided by an unreliable Pakistani news outlet as well as the claims of a local Taliban commander, which contradicted all other reports.

Here's the Bureau's reply:

His comments appear to refer to a CIA drone strike on February 9 in which local Taliban commander Badar Mansoor died. Citing just four sources, NAF's data reports only that three to five 'militants', including Mansoor, died in the attack. Among 18 unique sources we cite, the Bureau links to a story by Reuters, the international news agency. Reuters notes a Taliban commander's claims that Mansoor's wife and child died in the February 9 attack. Local paper The News also reported that Mansoor's wife and children were either injured or killed; and a Bureau field researcher reported anecdotal claims from the town that some of the leader's family had died.

As the Bureau notes, these overt claims of civilian deaths on February 9 remain contested. We state that between zero and two civilians reportedly died in the strike. It is not clear either way. What cannot be stated is that no civilians died.

The Bureau goes on to assert that there are specific, uncorrected errors in the New America Foundation data:

When the Bureau began looking in earnest at US drone strikes in summer 2010, we started to work with NAF's data, and that of the Long War Journal. At that time we had no interest in the time-consuming (and expensive) effort of compiling and maintaining accurate data on covert US strikes.

But the more we worked with NAF's material, the more troubled we became. In February 2011 for example, the Bureau wrote to NAF noting a number of errors. We pointed out a strike that it had missed entirely (November 5 2005). The Bureau also drew NAF's attention to a number of date errors. The Foundation claimed a strike had taken place on May 14 2005, for example. In fact that attack took place on May 8th. Bergen personally acknowledged the email, saying 'thanks for drawing attention to these.' Yet almost 18 months on those errors - easily verifiable - remain uncorrected.

Despite these alleged errors, the New America Foundation data is a valuable resource shedding light on a secretive military campaign that is very difficult to report on thoroughly and accurately. What's needed is greater awareness of its limits. It cannot tell us how many drone strikes occurred in a given year - only the approximation of how many were reported in the media. It cannot tell us how many "militants" were killed, how many civilians were killed, or the ratio between them. It can only tell us the minimum number of civilians reported to have been killed in the media. It cannot tell us whether the rate of civilian deaths has gone down -- only that the anonymous Pakistani officials relied upon for the majority of press reports say there are fewer civilian deaths. What if the Pakistanis feeding information to Western journalists have changed the way that they classify someone a militant in much the same way that President Obama did?