The Necessary Imperfection of Early Primary Polling

A number of factors makes it difficult to predict outcomes based on the numbers.

The conventional wisdom going into the Iowa caucuses held that, if voter turnout was high, Donald Trump would take the state. The Real Clear Politics average of polls showed a solid lead, the “gold standard” of Iowa polls said the same, and his closest competitor, Ted Cruz, wasn’t as popular in the state as before. But what actually happened Monday night confirmed another, perhaps quieter suspicion among primary-cycle observers: that Cruz’s ground game, and greater support among evangelicals, would carry the day.

For those shocked by the GOP results, take heart: Iowa is notoriously fickle, and not even top pollsters firmly predicted a win for Trump or Cruz, despite assumptions to the contrary. And it’d be wise to steel yourself for more unpredictability in the minutes, hours, and days leading up to next week’s primary in New Hampshire.

“We really do tend to look at [polls] as if they are somehow predictive...The best you can do is approximate and, if you’re a pollster, hope that the trends don’t shift on you,” David Redlawsk, the director of the Rutgers-Eagleton Poll, said with a laugh. “But they often do.”

When pollsters talk about polls—their own or others’—they’re always quick to offer a litany of warnings: Surveys have margins of error for a reason, to account for sampling mistakes; methodologies can vary wildly, depending on the pollster; and polls are just  “snapshots” of voter sentiment in time. For some, uncertainty keeps the election season exciting. “The reality is, we release our poll, stand by our method as our best shot, then obsess about how things could change on caucus night,” said J. Ann Selzer, the Des Moines Register polling maven who runs that “gold standard” survey, in an email. “If you are a political junkie, interested in how people make decisions in this democracy, this is all part of the thrill!”

Pollsters add that laymen might be overemphasizing the less meaningful parts of these surveys anyway. Their value is the insights they lend into “understanding voter priorities, concerns, tensions, [and] divisions” at a particular period of time, said Michael Dimock, the president of Pew Research Center, whose organization doesn’t do state-level polling. Those insights are helpful to understanding a race: If they weren’t available, he said, it’d be left up to journalists and the candidates themselves to report how people are feeling, and they’d be doing so anecdotally and through their own interpretation, respectively. (In a race where a hyperbolic candidate like Trump dominates the airwaves, having data to check the bluster is worthwhile.) But what polls can’t do is show with “a high degree of accuracy” the percentage of voters each candidate will snag in the end. “That’s just a level of prediction that they’re not designed to provide,” Dimock said.

Rewind to Monday in Iowa. Selzer said the “dearth of polls” showing a Cruz victory “is not a polling methods question”—a matter of survey set-up—but rather “a matter of the dynamic nature of the Iowa caucuses.” For one, voters in Iowa are allowed to register for the caucuses day-of. Redlawsk suggests pollsters might have simply caught Cruz at a bad time the week leading up to the caucuses, when Trump was hitting him on his Canadian birthplace, and he had a lackluster debate performance. Once most pollsters stopped their surveys ahead of Iowa, the winds could’ve shifted his way. Selzer notes that the caucuses are “built for people to decide late or change their minds.”

That’s not to say Cruz’s potential was totally lost on pollsters, though some news reports billed his win as a surprise or an upset over Trump. It’s true that Cruz had taken a recent dip in surveys, and the Register poll showed Trump with a lead: 28 percent to Cruz’s 23 percent, with a margin of error of 4 points. A Quinnipiac University poll surveyed voters up to the day before the caucuses, and it, too, showed Trump with an edge. But not only was Trump’s lead not wide enough to be definitive—leading some observers to emphasize Trump would need big turnout to win—but the Register itself suggested that “there’s still a strong case for Cruz in this race,” and called him “stronger than Trump, if you look beyond the horse-race numbers.” Peter Brown, the assistant director of Quinnipiac’s poll, notes that the percentage of voters Cruz eventually took, 27.7 percent, was only “minimally outside” the margin of error for his survey—a sign that the sampling was largely spot-on, and thus Cruz’s win shouldn’t have been a big shock. Brown notes, too, that the poll highlighted Rubio’s potential to challenge for second, and on Monday night, the Florida senator, though finishing third, carved out a greater slice of the electorate than many expected.

Overall, Redlawsk said, pollsters constantly struggle with the public’s tendency, “when it comes down to it, to ignore the nuance” in polling. They may’ve looked at Trump’s lead in Iowa and assumed that would be the end of it, even if the pollsters, and their numbers, framed the race differently.

Surveying for New Hampshire’s contest next week is subject to similar dynamics as in Iowa, where timing matters a great deal and where voters’ decision-making can be constantly in flux. It likely doesn’t help that the caucuses this year have been a big winnower of the field: Mike Huckabee and Martin O’Malley suspended their campaigns­­­­­­ the night of the caucuses, and Rand Paul and Rick Santorum did the same Wednesday. Their supporters in New Hampshire will be looking for alternatives in the next few days, or perhaps even decide not to vote. As primary-watchers look for clues as to how the race will shake out, pollsters would caution them to remember that surveys aren’t predictive and shouldn’t be taken as such.

That’s a hard truth to swallow for voters looking for certainty and journalists looking to play prophet. It might also be tough to swallow for a candidate like Trump, who’s put so much stock in his numbers. But maybe, just maybe, even he’s starting to get it: In a Tuesday night speech in New Hampshire, where he’s leading by a healthy amount, Trump observed that his Iowa stats had “maybe built a false expectation” of victory.