A propos of my post last week on asthma inhalers, an academic who asked not to be named wrote:
Maybe you should elaborate for some of your readers on the difficulty of arguing for null effects. In my experience, when people trying to get peer-reviewed science pubs arguing for null effects they do multiple experiments or the strongest test possible (which seems most relevant here). This study seems to have done neither and instead subjected all inhaler patients to crappier inhalers by showing that the mildest sufferers of asthma did not show a statistically significant difference in their relatively insensitive test.
This is a hugely important point. And there are a lot of ways in which these tests seemed deliberately designed to "prove" that there was no difference:
- Small sample: the smaller the sample, the harder it is to find an adverse effect. That's why drugs like Vioxx made it to market: distinguishing problems from background statistical noise needed a lot of patients. I know more than one analyst who argues that medical studies are generally too small--because humans are so variable, they don't reliably pick up any but the strongest effects. Hence the steady stream of articles proving that antioxidants will kill you/make you live forever/make you fat/make you thin/improve your singing voice/cause your fingers to fall off.
- Short timespan. One study ran for a year. The rest were 6-8 weeks.
- Only mild-to-moderate asthmatics included. These asthmatics are generally well controlled, and don't have crises that often. If you have a 20% increase in the number of crises over a year, but the asthmatics in your study only have a crisis once a week, it will be hard to distinguish that from statistical noise, espeically given the small samples and short timeframes.
- The differences may be hard to quantify, and thus not show up in the study: if your breathing gets 30% worse, the doctor can't tell unless he happens to have you on hand to measure when you're having an attack. Again, if you only have an attack rarely, he probably won't.
A cursory look at some of the studies indicates that they didn't really show there's no difference; what they showed was that there was no difference that a) showed up in a lab in b) a small sample of c) the patients with the mildest disease over d) generally short timeframes.
Lest you think this is special pleading, I'm pretty much resigned to my CFC-fate. But this sort of thing matters broadly. The FDA used the lightest possible statistical test on a pretty important medication for millions of asthmatics. Do you want Medicare denying your mother a possibly effective treatment for her otherwise terminal cancer with the same kind of test?