New Hampshire Polling: What Went Wrong?

The American Association for Public Opinion Research (professional consortium of pollsters) has published a detailed report on polling in the 2008 primaries, inspired by the New Hampshire debacle. As many readers will recall, polls leading up to the New Hampshire primary had Barack Obama ahead by an average of eight percentage points, while some major polls had him up more than 10; Hillary Clinton ended up winning by 2.6 percent. (Click here for a chronological chart of New Hampshire polls, compared to the final result.)

This roiled the political world, leaving everyone with the question: how could the pollsters have been so wrong?

The AAPOR report finds that the early primary date, underrepresentation of Clinton-supporting groups, an influx of first-time voters, and the race of survey interviewers could have caised the New Hampshire errors. From the report:

Factors that may have influenced the estimation errors in the New Hampshire pre-primary polls include:
1. Given the compressed caucus and primary calendar, polling before the New Hampshire primary may have ended too early to capture late shifts in the electorate there, measuring momentum as citizens responded to the Obama victory in the Iowa caucus but not to later events in New Hampshire.
2. Patterns of nonresponse, derived from comparing the characteristics of the pre-election samples with the exit poll samples, suggest that some groups that supported Senator Hillary Clinton were underrepresented in the pre-election polls.
3. Variations in likely voter models could explain some of the estimation problems in individual polls. Application of the Gallup likely voter model, for example, produced a larger error than their unadjusted data. While the "time of decision" data do not look very different in 2008 compared to recent presidential primaries, about one-fifth of the voters in the 2008 New Hampshire primary said they were voting for the first time. This influx of first-time voters may have had an adverse effect on likely voter models.
4. Variations in weighting procedures could explain some of the estimation problems in individual polls. And for some polls, the weighting and likely voter modeling were comingled in a way that makes it impossible to distinguish their separate effects.
5. Although no significant social desirability effects were found that systematically produced an overestimate of support for Senator Obama among white respondents or for Senator Clinton among male respondents, an interaction effect between the race of the interviewer and the race of the respondent did seem to produce higher support for Senator Obama in the case of a black interviewer. However, Obama was also preferred over Clinton by those who were interviewed by a white interviewer.

The AAPOR committee broadly indicts polling firms for not disclosing their New Hampshire methodologies "in a timely manner," leaving its researchers without sufficient information. As a result, some of the more interesting theories could not be studied empirically:

Finally, factors that appeared to be potential explanations for estimation errors, but for which the committee lacked any empirical information to assess include:
1. Because of attempts by some states to manipulate the calendar of primaries and caucuses, the Iowa and New Hampshire events were rescheduled to the first half of January, with only five days between the events, truncating the polling field period in New Hampshire following the Iowa caucus.
2. The order of the names on the ballot - randomly assigned but fixed on every ballot - may have contributed to the increased support that Senator Hillary Clinton received in New Hampshire.

So that's the answer--or at least as good an answer as we're likely to get.