Author of Iraq body count study censured by a group he doesn't belong to

Gilbert Burnham, the lead author of two papers published in The Lancet on civilian casualties in Iraq, has been censured by the American Association for Public Opinion Research for failing to provide adequate details about the survey.

I don't know what to make of this.  Burnham is not a member of the AAPOR, so I'm not sure why they felt they had to censure him.

On the other hand, I heard enough during my work on a story about the body count to know that the Hopkins team was behaving in odd ways that dramatically piqued the suspicions of their critics.  They made claims about their methodology, and then retracted them.  They kept very close control over their data set, distributing it to only a few people, and refusing to release even well scrubbed data to almost anyone who disagreed with them.   On top of that, they put limitations on what researchers who had seen the data could say to the media.  I repeatedly asked for what seemed to me to be completely anodyne pieces of information in order to investigate claims of "curbstoning" and other polling offenses, only to be told that the researchers who had access to the dataset were not allowed to give out any specific numbers.

The team's defenders said that they had to do this either for the security of the responders (even though no one was asking for the raw data with identities attached), or because the people who wanted it were too stupid or disagreeable to be trusted with the data set.  AAPOR's position, which I think is right, is that this is not the way the scientific method works.  To the extent that this is, as some have claimed, standard practice in the public health community, it does not validate the behavior; it casts serious doubt on the output of the field.

I, too, may have encountered the Burnham stonewall.  I happened to be writing my story just as the World Health Organization study that was highly critical of Burnham, et. al. was released. Les Roberts, who had become the public face of the team, was making frankly lunatic claims on the radio that the two studies basically agreed, even though the introduction to the WHO study specifically said that their results made it very unlikely that Burnham et. al. had been correct.  This claim was so unusual that when I asked neutral conflict epidemiologists, they patiently explained that I couldn't possibly have heard Roberts correctly, because no one with half a brain would ever have said that.

With controversy swirling, I called the Hopkins PR office, only to be told that Burnham was off in Amman for work and couldn't be reached for comment.  Given that must have already known that the report was due to be released, and roughly what it was going to say--I was told about the report several days before its release, by another journalist--this seemed very strange.  Stranger was the Bloomberg School of Public Health's apparent total lack of interest in explaining the study to a journalist from a major national magazine. I asked them to call me when he was available, but apparently his unavailability covered the entire month of January, and beyond.

Perhaps this was because the National Journal, my sister publication, had just published an article that was highly critical of the study.  Or maybe they just forgot.  But the general tendency to refuse to engage with anyone who seemed likely to question their findings did not inspire confidence in the results.

Update  Mark Blumenthal notes that Tim Lambert got a response from Mary Losch on their specific concerns:


I have read your entry and would note that the links you provided did not supply the questionnaire items but rather a simple template (as noted in the heading). The Johns Hopkins report provides only superficial information about methods and significantly more detail would be needed to determine the scientific integrity of those methods -- hence our formal request to Dr. Burhnam. The Hopkins website refers to data release but, in fact, no data were provided in response to our formal requests. Included in our request were full sampling information, full protocols regarding household selection, and full case dispositions -- Dr. Burnham explicitly refused to provide that information for review.

We do not provide public reports of the investigations but if there are other specific questions that I could answer, I would be happy to try to do so.

I also asked Losch why AAPOR considers the "template" of questions posted online to be something less than "the wording of the questions used." She replied that they "requested the survey instrument, (including consent information) and it was not provided. The template did not appear to be much beyond an outline and certainly was not the instrument in its entirety."

Still no word on why the AAPOR got involved in the first place.  But I don't see any reason to think that they're part of the scientifically illiterate and politically motivated conspiracy against Burnham et al. that the team's supporters have alleged.

Update 2  Just to be clear:  I have no reason to think that Burnham or any of the Hopkins team committed knowing fraud, as some have alleged.  I don't know that there is anything wrong with their data.  But the secrecy seems bizarre and wholly unnecessary, which makes it harder to trust their results.