Comparative Effectiveness Redux

A propos of my post last week on asthma inhalers, an academic who asked not to be named wrote:

Maybe you should elaborate for some of your readers on the difficulty of arguing for null effects. In my experience, when people trying to get peer-reviewed science pubs arguing for null effects they do  multiple experiments or the strongest test possible (which seems most relevant here). This study seems to have done neither and instead subjected all inhaler patients to crappier inhalers by showing that  the mildest sufferers of asthma did not show a statistically significant difference in their relatively insensitive test.

This is a hugely important point.  And there are a lot of ways in which these tests seemed deliberately designed to "prove" that there was no difference:

  • Small sample:  the smaller the sample, the harder it is to find an adverse effect. That's why drugs like Vioxx made it to market:  distinguishing problems from background statistical noise needed a lot of patients.  I know more than one analyst who argues that medical studies are generally too small--because humans are so variable, they don't reliably pick up any but the strongest effects.  Hence the steady stream of articles proving that antioxidants will kill you/make you live forever/make you fat/make you thin/improve your singing voice/cause your fingers to fall off.
  • Short timespan.  One study ran for a year.  The rest were 6-8 weeks.
  • Only mild-to-moderate asthmatics included.  These asthmatics are generally well controlled, and don't have crises that often.  If you have a 20% increase in the number of crises over a year, but the asthmatics in your study only have a crisis once a week, it will be hard to distinguish that from statistical noise, espeically given the small samples and short timeframes.
  • The differences may be hard to quantify, and thus not show up in the study:  if your breathing gets 30% worse, the doctor can't tell unless he happens to have you on hand to measure when you're having an attack.  Again, if you only have an attack rarely, he probably won't.

A cursory look at some of the studies indicates that they didn't really show there's no difference; what they showed was that there was no difference that a) showed up in a lab in b) a small sample of c) the patients with the mildest disease over d) generally short timeframes.

Lest you think this is special pleading, I'm pretty much resigned to my CFC-fate.  But this sort of thing matters broadly.  The FDA used the lightest possible statistical test on a pretty important medication for millions of asthmatics.  Do you want Medicare denying your mother a possibly effective treatment for her otherwise terminal cancer with the same kind of test?

The most worrying thing here is the real possibility that the FDA got the result the EPA wanted.  Will they be tempted to get the answer Medicare would like to hear about the relative merits of expensive medications?

Again, I'm not saying we shouldn't do CRE.  But for all that Democrats are enjoying thinking of themselves as the Party of Guys in White Coats With The Answers, the binary discussion of CRE (we'll find what works!) is borderline religious in the way it treats government researchers.  The process of finding out what works is considerably more complicated than giving a scientist some money and a hundred human lab rats.  And there is a real danger that a few studies will end up shutting down potentially useful treatments, as first Medicare and then private insurers turn weak or equivocal results into an iron ruling.

Presented by

Megan McArdle is a columnist at Bloomberg View and a former senior editor at The Atlantic. Her new book is The Up Side of Down.

Saving the Bees

Honeybees contribute more than $15 billion to the U.S. economy. A short documentary considers how desperate beekeepers are trying to keep their hives alive.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

How to Cook Spaghetti Squash (and Why)

Cooking for yourself is one of the surest ways to eat well.

Video

Before Tinder, a Tree

Looking for your soulmate? Write a letter to the "Bridegroom's Oak" in Germany.

Video

The Health Benefits of Going Outside

People spend too much time indoors. One solution: ecotherapy.

Video

Where High Tech Meets the 1950s

Why did Green Bank, West Virginia, ban wireless signals? For science.

Video

Yes, Quidditch Is Real

How J.K. Rowling's magical sport spread from Hogwarts to college campuses

Video

Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

More in Business

Just In