Department of Awful Statistics

More

Bill Easterly has a good post on bad infant mortality stats:

Of the 193 countries covered in the study, the researchers were able to use actual, reported data for only 33. To produce the estimates for the other 160 countries, and to project the figures backwards to 1995, the researchers created a sophisticated statistical model. [1]

What's wrong with a model? Well, 1) the credibility of the numbers that emerge from these models must depend on the quality of "real" (that is, actual measured or reported) data, as well as how well these data can be extrapolated to the "modeled" setting ( e.g. it would be bad if the real data is primarily from rich countries, and it is "modeled" for the vastly different poor countries - oops, wait, that's exactly the situation in this and most other "modeling" exercises) and 2) the number of people who actually understand these statistical techniques well enough to judge whether a certain model has produced a good estimate or a bunch of garbage is very, very small.

Without enough usable data on stillbirths, the researchers look for indicators with a close logical and causal relationship with stillbirths. In this case they chose neonatal mortality as the main predictive indicator. Uh oh. The numbers for neonatal mortality are also based on a model (where the main predictor is mortality of children under the age of 5) rather than actual data.

So that makes the stillbirth estimates numbers based on a model...which is in turn...based on a model.

In many parts of the world, data is hard to come by.  Unfortunately, voters and donors demand data . . . and when it can't be collected effectively, researchers under heavy pressure to come up with numbers are forced to use alternative methods.

But publishing a number is dangerous: the caveats will be stripped off by an innumerate media, and even if they were left in, the public won't understand what they mean.  When the quality of the data is really bad, the public is left less informed than it was before the number was published.  I wrote about that problem in regard to Iraq mortality statistics a few years ago, but it's much broader than conflict epidemiology.  At some point, we're better of knowing that we don't know.
Jump to comments

Megan McArdle is a former writer and editor at The Atlantic.

Get Today's Top Stories in Your Inbox (preview)


Elsewhere on the web

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register. blog comments powered by Disqus

Video

Miami: The Next Big Start-Up City?

How the city became a center for innovation

Video

Video

A Brief History of Romantic Comedies

From The Atlantic's Chris Orr

Video

Video

Life in 'the New Arctic'

A moving portrait of a fading landscape

Video

Video

The Rise of New York City

A fascinating look at Manhattan in the 1940s

Video

'I Thought It Was Really Funny, but No One Else Did'

A day with New Yorker cartoonist Joe Dator

Video

New Yorkers: The Winemaker

Make your own wine ... in New York City

Video

What Is Methane Hydrate?

"Flaming ice" is a vast natural energy source

Video

NASA's Time-Lapse of the Sun

Now with epic dubstep music

Video

A Video Letter From the Editor

Highlights from the May 2013 issue

Video

Shaken Not Tuned: Cocktail Experiments

Can a tuning fork improve a cocktail?

Video

Video

The Rise of Environmentalism

Tracking 50 years, from the Love Canal disaster to Greenpeace

Video

Is He Cheating? A 1950s Guide

'That little blonde secretary from the office?’

Video

New Yorkers: Vintage Vacuum-Tube Amps

Risking electric shock to restore old amplifiers

Video

The DIY Piano-Bicycle

Everybody needs a hobby

Writers

Up
Down

More in Business

In Focus

2013 National Geographic Traveler Photo Contest