Did RomneyCare Really Have No Effect on Medical Bankruptcy in Massachusetts?

My favorite medical bankruptcy research team has gone back to the well one more time, with a study that assesses the change in medical bankruptcies in Massachusetts since health care reform passed there.  Elizabeth Warren is no longer on the team for some reason, perhaps because her work for the government prohibited her from participating.  But Steffie Woolhandler and David Himmelstein, stalwarts of Physicians for a National Health Program, are still there.  So some people may be surprised at what they find:  health reform in Massachusetts has not decreased the rate of medical bankruptcies very much--indeed, the modest drop they did find wasn't statistically significant.

Just to recap the history of this team's work, which I blogged about here in more detail: they are now the nation's leading providers of eye-popping statistics suggesting that large majorities of all bankruptcies are medical.  (This is the third in a series).  They get this result by using extremely generous definitions of what constitutes a "medical bankruptcy", which generates a rate of "medical bankruptcy" about twice the rate that you get by . . . um . . .  asking people why they declared bankruptcy.  In their first study of 2001 bankruptcies, published in Health Affairs in 2005, these were the criteria:

We also used the questionnaire to estimate how frequently illness and medical bills contributed to bankruptcy. We developed two summary measures of medical bankruptcy. Under the rubric "Major Medical Bankruptcy" we included debtors who either (1) cited illness or injury as a specific reason for bankruptcy, or (2) reported uncovered medical bills exceeding $1,000 in the past years, or (3) lost at least two weeks of work-related income because of illness/injury, or (4) mortgaged a home to pay medical bills. Our more inclusive category, "Any Medical Bankruptcy," included debtors who cited any of the above, or addiction, or uncontrolled gambling, or birth, or the death of a family member.
This found that slightly over half of all bankruptcies were "medical bankruptcies", while slightly under half were "major medical bankrupties".

In 2009, they published a follow-up study of 2007 bankruptcies using slightly more restrictive criteria, but also comparing their results to the earlier study:

For this analysis, we replicated the most conservative de- finition employed in the 2001 study, which designated as "medically bankrupt" debtors citing illness or medical bills as a specific reason for bankruptcy; OR reporting uncovered medical bills $1000 in the past 2 years; OR who lost at least 2 weeks of work-related income due to illness/injury; OR who mortgaged a home to pay medical bills. Debtors who gave no answers regarding reasons for their bankruptcy were excluded from analyses.

For all other analyses (ie, those not reporting time trends) we adopted a definition of medical bankruptcy that utilizes the more detailed 2007 data. We altered the 2001 criteria to include debtors who had been forced to quit work due to illness or injury. We also reconsidered the question of how large out-of-pocket medical expenses should be before those debts should be considered contributors to the family's bankruptcy. Although we needed to use the threshold of $1000 in out-of-pocket medical bills for consistency in the time trend analyses, we adopted a more conservative threshold--$5000 or 10% of household income--for all other analyses. Adopting these more conservative criteria reduced the estimate of the proportion of bankruptcies due to illness or medical bills by 7 percentage points

These results were even more startling; under the old definition, medical bankruptcies had increased to about 70% of all bankruptcies, while under the new criteria, they were still about 62%.

Their defenders have argued that they were simply trying to isolate the bankruptcies where illness contributed, not where it was causal.  But there are a few problems with this.  First of all, their definition picks up bankruptcies where medical bills aren't causal, since people who are going bankrupt throw in all their unsecured debts whether they're problematic or not.  As I pointed out at the time, by this definition, MIchael Vick had a "medical bankruptcy"; so did Patty Barreiro, who I wrote about here.  Like Vick, the clear causes of her bankruptcy were  consumer debt and a legal judgement, but her filing also included thousands of dollars worth of unpaid bills to what appears to be a dermatology and cosmetic medicine practice.

Second of all, the term "medical bankruptcy" is going to be picked up by the media and policymakers as causal, rather than "contributory" or incidental.  The media also, naturally, focuses on medical bills rather than income loss, even though their method is designed to pick up a lot of problems caused by income shocks rather than medical bills.  Not their fault that they're misinterpreted?  Read Steffie Woolhandler's testimony to Congress in the run-up to health care reform.

Between 2001 and 2007, the proportion of all bankruptcies attributable to medical problems rose by nearly half. In order to compare the medical bankruptcy rates in 2007 and in our 2001 study we had to use the same definitions in both years. Our 2001 study had used a less stringent ("legacy") definition of medical bankruptcy that included families with more than $1000 in unpaid medical bills. Using this "legacy" definition, the medical bankruptcy rate rose from 46.2% in 2001 to 69.1% in 2007 - a 49.6% increase. This is clear evidence that health care is becoming less affordable to American families, including American families with health insurance.
Woolhandler has left out what I think is a rather central fact--one that I blogged about at the time.  Namely, that the actual number of "medical bankruptcies" had fallen dramatically since 2001--the proportion had risen, but the number of bankruptcy filings had dropped off a cliff thanks to the effects of the 2005 bankruptcy reform. This is, to put it mildly, rather a problem for the thesis that rising medical bills are pushing ever-more families to the edge of financial disaster. Their paper also goes to rather great lengths to avoid mentioning this--dealing only in percentages, but never mentioning the absolute numbers. Interviews she's given to the media show a similar emphasis.

So: is this new paper any better?

Well, at least this time they've found room to include the absolute numbers as well as the percentages, perhaps because this time, the absolute numbers favor their argument:

Despite a marked declined in the uninsurance rate in Massachusetts since the implementation of health reform, the proportion of bankruptcies that occurred in the wake of medical problems has not decreased significantly, and the absolute number of medical bankruptcies has actually increased by one third. The deep recession beginning in 2008 surely played an important role in increasing the bankruptcy rate and left many families more vulnerable to financial shocks from illness. However, our findings are incompatible with claims that health reform has cut medical bankruptcy filings significantly
However, this paper commits the same error as their earlier paper, in the opposite direction. The earlier paper didn't adequately address the fact that the 2005 bankruptcy reform had depressed bankruptcies, reducing the absolute number and complicating their argument considerably (aside from the fact that the number of "medical" bankruptcies fell, we have the problem of whether medical bankruptcies may have fallen at a different rate from other bankruptcies.  They do some handwaving about how there's no evidence that they did--but they reference a paper that doesn't really support their argument, making their rebuttal less than convincing.)

Bizarrely, this paper doesn't mention the bankruptcy reform at all, even though absolutely everyone agrees that in 2007, bankruptcies were still rebounding from an abnormally low level, because so many people had rushed to file before the end of 2005, when the new law took effect.  To speak of the total number of medical bankruptcies rising by a third without offering this context is misleading in the extreme.  And I've argued before that even discussing the proportion without referencing the 2005 changes is misleading because it's very possible that medical bankruptcies were less affected by the change than other forms of bankruptcy--either job loss or medical bills from a sudden health crisis are arguably harder to see coming than a business failure, a divorce, or simply the cumulative results of years of living up to and beyond one's income.  If that's so, that could account for the entirety of the small (not statistically significant) reduction that they found in medical bankruptcies in Massachusetts.

Beyond the same problems that have plagued their work for years, there's another issue.  In order to assess the effects of health care reform in Massachusetts, they compare their new survey results to the 2007 study.  But the 2007 study was a nationwide sample.  There are all sorts of problems with this, but the most glaring one is that they're comparing their current study of 199 filers to an earlier group of only 44.  To be sure, the number of bankruptcy filings in Massachusetts is quite low--only 13,000 in 2007, and about 23,000 last year.  Nonetheless, 44 cases still seems pretty thin to me.

I expect that over the next few days, you'll see more than a few conservatives trumpet this result.  Even the liberal Physicians for a National Health Program . . . the citations will begin.

But this is simply not good support for anyone's argument.  First of all, this is not the research equivalent of a "statement against interest":  Woolhandler, at least, was vociferously opposed to making private insurers part of health care reform, so it's not some shocking departure for her to publish a paper arguing that a private-insurer-based model is essentially little better than nothing.

More importantly, the data quality is extremely low.  The paper doesn't account for obvious confounding factors.  It uses a very small sample for comparison, one that wasn't really selected to assess state-level effects. And by using an overly broad definition of medical bankruptcies, it practically guarantees that reform will show little effect.  Health care reform won't keep a plumber who's been hit by a car at work instead of in the hospital.  It isn't going to mean that nobody ever has an unpaid medical bill at the time of declaring bankruptcy.  It doesn't pay the mortgage if Mom has to quit work to take care of a kid with autism.  Canada still has medical bankruptcies, despite single payer, because medical problems cause income loss as well as higher medical bills.

It's not that this is an entirely unbelievable result (they did find a decent-size drop, after all; it's just that because their sample was so small, it wasn't statistically significant).  But this paper, like the ones before it, is simply too badly flawed to rely on.  Their work wasn't a good argument for health care reform (for all that it was used that way, over and over).  It isn't a good argument for repeal, either.

Update (3 pm):  Avik Roy has more.