Is It Evidence? Is it Medicine?

Daniel Davies voices some of my own reservations about "evidence based medicine":

And so, NICE has decided, on the basis of "the evidence", that acupuncture and chiropractic are a good way to spend the NHS's money. Except when you look at it, "the evidence" isn't really all that great. As Edzard Ernst points out, the Cochrane Institute (the other great temple of evidence-based medicine) actually found chiropractic to be more or less useless, while the evidence for acupuncture is that all of the ancient wisdom and theory of the meridians and qi doesn't actually confer any great benefit over and above that which can be gained from simply lying on a table and being poked with sticks.

Part of the problem is that in the specific case of lower back pain, it's a notoriously difficult condition to understand or treat, and a lot of the art is simply to find a nice and professional-sounding way of saying "live with it, there's nothing we can do" that doesn't make the patient give up hope and suffer even more. But another part of the problem is that the overall assessment of what "the evidence" was, was made by a committee that had a bunch of spinal manipulation enthusiasts on it.

Which brings me to the problem; this is exactly what we should have expected, and it's the reason why I've been putting the phrase "the evidence" in great big scare-quotes. Because the actual medical evidence on lower back pain isn't something that can be nicely summarised in a slim paper guideline; it's spread out across millions of individual lower backs, some fraction of the experiences of which are summarised into hundreds of research papers, which were then distilled down into the Cochrane review, which was itself processed through the NICE committee. Basically what evidence-based medicine is about, at this level, is somebody making a decision about what the facts are going to be.

And if that decision about "what the facts are" is one that is going to determine the handing out of large chunks of government cash, then you bet that the enthusiasts of every theory there is are going to move hell and high water to get themselves on that committee. Not out of any venial motive, but because they believe in their theory, and a contrary NICE guideline has the potential to kill it stone dead. So what happens is that the process of finding out the underlying truth, which is of necessity slow, unclear and often completely open-ended, gets accelerated and politicised. It's what you might call "government science".

It's a phenomenon that's very familiar to economists under the name "Goodhart's Law". Basically, Goodhart's Law says that "any economic relationship which is used for policy purposes, ceases to be valid". In other words, you can have an economic model which works tolerably well as an understanding of how, say, the relationship between money, prices and output works. But when you try to use that model to set interest rates, then suddenly the model itself is part of the recursion - part of the system that you're trying to control - and this changes the nature of the relationship that you were trying to use.

Similarly, in the early days of the evidence-based medicine movement, when they were the Young Turks or punk rockers, shaking up a complacent medical establishment that had got out of touch with the cutting edge of medical research, they had the potential to do a lot of good. But now they are the establishment, and as a result of that, the very evidence that they rely on, is shaped by the fact that it needs to appeal to them. The fact that a movement which begun by trying to bring science back into medicine, has now ended up putting its imprimateur on some obvious pseudoscience, ought to worry us more than it does, because this is only the most obvious manifestation of the general problem.

This is part of a broader problem with medicine and other sciences with physics envy.  Medicine, like economics, is really messy.  You can't do the same kind of controlled experiments that you can do on rats or quarks, and as a result, the results are often hard to interpret.  But this doesn't stop doctors, or policymakers, from acting as if the studies or metastudies can deliver vastly more certainty than is possible from such inherently sloppy science.  This is why, for example, I am broadly sympathetic to Paul Campos' claim that medical guidelines on obesity tell you much more about the attitudes towards fat in the upper middle class social stratum that doctors occupy, than about reliable scientific evidence on same.

But policy demands certainty.  And so you get obesity guidelines advising everyone to diet and excercise to shed their excess pounds, even though it's as close to a scientific certainty as anything is that most people simply regain any weight they manage to diet off.  And you get absurdly precise economic forecasting, even though in many cases, the better answer would be "who knows?"

In both cases, I don't see a better alternative.  But we should be more skeptical of both the institutions, and their claims.