Earlier this week, the Pew Research Center released a report examining several screening methods, including cutting-edge screens using computer algorithms and machine learning. Using data from the 2014 midterm elections, researchers compared how a person responded to pre-election questions against their actual 2014 voting record—a luxury pollsters didn’t have at the time.
The report’s conclusion: The old ways leave much to be desired, especially for unusual elections.
“These methods, which have been around for so long, may be losing some of their accuracy because circumstances have changed,” said Scott Keeter, a senior survey adviser at Pew Research. “Whether there has been a change in our politics in just the last two years that makes all of this less accurate is really impossible to answer at this point.”
For years, the gold standard in screening likely voters was a set of questions developed in the 1950s by Gallup statistician Paul Perry. Recognizing that simply asking people if they planned to vote wasn't enough—nearly everyone says yes—Perry posed a series of questions that factored into a final score.
Here’s Pew’s version:
- How much thought have you given to the coming November election?
- Have you ever voted in your precinct or election district?
- Would you say you follow what’s going on in government and public affairs most of the time, some of the time, only now and then, hardly at all?
- How often would you say you vote?
- How likely are you to vote in the general election this November?
- In the 2012 presidential election between Barack Obama and Mitt Romney, did things come up that kept you from voting, or did you happen to vote?
- Please rate your chance of voting in November on a scale of 10 to 1.
Tallying up these questions, pollsters would estimate the expected turnout for the election and cut the list accordingly. Only the most engaged voters would be included—or so the theory went.
But Pew’s research shows a number of these questions have dubious predictive value. For instance, pollsters will routinely drop respondents who say they have little interest in politics or haven’t followed the election. However, 55 percent of the people in Pew’s study who said they thought “only a little” about the coming election ending up voting on Election Day.
"We found in 2012 that a lot of people said, 'I have no interest in this election, but you can be damn sure I’m going to show up to vote,’“ Keeter said, recounting individual conversations with voters. "That showed to me that question might not be as useful."
Not even machine learning, the magic elixir of our time, can completely solve this problem. Keeter and his crew built several models to predict 2014 voter turnout using logistic regressions and the "random forest" algorithm, which slices and dices data to find correlations difficult for humans to pick out. The results were decent, but still not spot-on.