Fixing the Pollster's Problem of Finding "Likely Voters" Is Harder Than It Sounds

A month before the 2000 presidential election, George W. Bush benefited from a statistical miracle.

He started October trailing opponent Al Gore, with a CNN/USA Today/Gallup poll projecting a 51-40 split among likely voters. Then came the first presidential debate—remember, the one with the sighing? A few days later, Bush was on top, 49-41, a stunning 19-point swing.

But wait! Gallup polling also showed most voters thought Gore had won the debate. So had Bush shot ahead—by losing?

Looking back, social scientists doubt public opinion was really bobbing around like a pogo stick. Instead, they blame the filtering Gallup used to determine “likely voters”—a vexing bit of psychological prediction pollsters have never gotten quite right.

If you’re in the business of guessing the future, even the best presidential polls do no good if they’re diluted by the opinions of people who will never make it to the ballot box. Studies show unengaged respondents skew Democratic, and polls that don’t filter for registered or likely voters will reliably underestimate Republican support.

Voter screens make polls more accurate. But they’re also a source of variation and error. With several recent high-profile whiffs in the polling world—see the Kentucky governor’s race in 2015, or several House races in 2014—they’re getting another look.

Earlier this week, the Pew Research Center released a report examining several screening methods, including cutting-edge screens using computer algorithms and machine learning. Using data from the 2014 midterm elections, researchers compared how a person responded to pre-election questions against their actual 2014 voting record—a luxury pollsters didn’t have at the time.

The report’s conclusion: The old ways leave much to be desired, especially for unusual elections.

“These methods, which have been around for so long, may be losing some of their accuracy because circumstances have changed,” said Scott Keeter, a senior survey adviser at Pew Research. “Whether there has been a change in our politics in just the last two years that makes all of this less accurate is really impossible to answer at this point.”

For years, the gold standard in screening likely voters was a set of questions developed in the 1950s by Gallup statistician Paul Perry. Recognizing that simply asking people if they planned to vote wasn't enough—nearly everyone says yes—Perry posed a series of questions that factored into a final score.

Here’s Pew’s version:

How much thought have you given to the coming November election?

Have you ever voted in your precinct or election district?

Would you say you follow what’s going on in government and public affairs most of the time, some of the time, only now and then, hardly at all?

How often would you say you vote?

How likely are you to vote in the general election this November?

In the 2012 presidential election between Barack Obama and Mitt Romney, did things come up that kept you from voting, or did you happen to vote?

Please rate your chance of voting in November on a scale of 10 to 1.

Tallying up these questions, pollsters would estimate the expected turnout for the election and cut the list accordingly. Only the most engaged voters would be included—or so the theory went.

But Pew’s research shows a number of these questions have dubious predictive value. For instance, pollsters will routinely drop respondents who say they have little interest in politics or haven’t followed the election. However, 55 percent of the people in Pew’s study who said they thought “only a little” about the coming election ending up voting on Election Day.

"We found in 2012 that a lot of people said, 'I have no interest in this election, but you can be damn sure I’m going to show up to vote,’“ Keeter said, recounting individual conversations with voters. "That showed to me that question might not be as useful."

Not even machine learning, the magic elixir of our time, can completely solve this problem. Keeter and his crew built several models to predict 2014 voter turnout using logistic regressions and the "random forest" algorithm, which slices and dices data to find correlations difficult for humans to pick out. The results were decent, but still not spot-on.

One surefire way to improve accuracy, Keeter says, is to match each respondent with their voting record, available via a public file that marks whether an individual voted in previous elections (though not how they cast their ballot). But that requires the cooperation of the poll respondents, who may understandably be unwilling to give their names and addresses to a stranger over the telephone.

And if a machine-learning approach takes off, another danger arises: the false confidence of the algorithm. A pollster could use a computer to make a beautiful model that accurately predicts how a voter would have acted in 2014. But would that have any relevance in the Trump-powered campaign of 2016?

Gallup has taken plenty of heat for its likely voter screens—perhaps most notably from Nate Silver, the founder of FiveThirtyEight. This time around, the polling company says it will sit out the 2016 primary horse race. But Pew’s research shows the challenge of measuring voter interest extends far beyond just one company.

Likely voter predictions dramatically improve the closer you get to an election. In the future, given polling’s other problems, that may be all we can reasonably expect.

About the Author

Andrew McGill is a senior product manager at The Atlantic.

A new study shows that families act on insufficient information when it comes to figuring out where to enroll their children.

A person trying to choose their next set of wheels might see that car A made it farther than car B in a road test and assume it gets better gas mileage. But that’s only true if the two tanks are filled with the same substance. Putting high-octane gas in one and water in the other, for example, provides little useful information about which car makes the most of its fuel. A new working paper titled “Do Parents Value School Effectiveness?” suggests that parents similarly opt for schools with the most impressive graduates rather than figuring out which ones actually teach best. The study joins a body of research looking critically at what it means for a school to be successful.

Take the work of Erin Pahlke, for example. The assistant professor of psychology at Whitman College saw research showing that girls who attend school only with other girls tend to do better in math and science. The trick, she said, is that those studies didn’t analyze “differences in the students coming into the schools.” As it turns out, those who end up in same-sex schools tend to be wealthier, start out with more skills, and have parents who are more proactive than students who attend co-ed institutions. In a 2014 meta-analysis, Pahlke and her colleagues reviewed the studies and found when examining schools with the same type of students and same level of resources—rather than “comparing [those at] the public co-ed school to [their counterparts at] the fancy private school that’s single-sex down the road”—there isn’t any difference in how the students perform academically. Single-sex schooling also hasn’t been shown to offer a bump in girls’ attitudes toward math and science or change how they think about themselves. In other words, it often looks like single-sex schools are doing a better job educating kids, but they aren't. It's just that their graduates are people who were going to do well at any school. They’re running on high-octane gas.