Nate Silver on How the Data Could Be Wrong

Nate Silver and his Fantastic Election Prediction Machine over at NY Times blog FiveThirtyEight have been modeling out a Republican takeover in the run-up to today's midterms, but Silver offers us a seed of doubt, explaining how polls and predictions could be wrong, and how Democrats could have a better night than expected.

Silver's methodology is the most mathematical of anyone who predicts elections for a living. At the Cook and Rothenberg political reports, the gurus look at plenty of data--polling (both internal and external) in hundreds of individual races, plus some national polling trends and extraneous wave-producing factors, to come up with a final number. Silver, meanwhile, has a more specified technique--a formula, in fact--that takes into account district polling plus a regression of other metrics where polling isn't as plentiful, along with national metrics to create weighted probabilities, which are then used to simulate the election over and over, finally producing an average outcome.

Currently, FiveThirtyEight shows the GOP picking up 55 House seats (they need 39 to take over) and 7 Senate seats (short of the needed 10).

But here are Silver's five reasons for why the polls and predictions could be wrong:

  1. The cellphone effect. ... this probably does not mean that Democrats are bound to overperform their polls by four or five points. A fair number of polls do include cellphones, so at best it might be half that. And the effects probably aren't so uniform from company to company. Still, this is a theory that has a fair amount of evidence behind it.
  2. The "robopoll" effect. ... Unlike in past years, there are significant differences between the results shown by automated surveys and those which use live human interviewers -- the "robopolls" being 3 or 4 points more favorable to Republicans over all, although the effects vary a lot from firm to firm.
  3. Some likely voter models, particularly Gallup's, may "crowd out" Democratic voters. ... The Gallup poll and the Gallup poll alone is probably responsible for much of the sense of impending doom that Democrats feel and the (premature for at least 24 more hours) sense of triumphalism that Republicans are experiencing. But there is quite a bit of room to critique the poll. The basic potential issue is that Gallup uses fixed turnout targets. For instance, they estimate that 40 percent of the electorate will vote, and then let their respondents fight it out to see who the 40 percent most likely to vote are.
  4. Democrats probably have better turnout operations. ... Democrats probably do have an edge in this department with the voter lists and infrastructure they built up during Barack Obama's campaign, and which have been perpetuated to some extent by Organizing For America. ... Democrats might be able to coax an extra percentage point or two of their vote to the polls, especially in states like Ohio and Pennsylvania where they've invested a ton of resources over the years.
  5. The consensus view of Democratic doom is not on such sound footing as it seems. ... all of these indicators are, in fact, highly correlated with one another. They're all rooted in the polling, and they're all dependent on the polling basically being accurate. There's not much diversity at all: it's just different manifestations of the same thing. ... The case that Democrats could do better than expected -- not well, by any means, merely better than expected -- rests a little more in the realm of what artists call negative space: not what there is, but in what there isn't. There aren't 50, or even more than about 25, districts in which Republican candidates are unambiguous favorites.

Read more at FiveThirtyEight.