-- Landline sampling frame: Gallup used a listed landline sample -- that is, a roster of landlines tied to actual residents -- resulting in an "older and more Republican" sample than what they might have compiled using a sample obtained by randomly dialing landline telephones. "These differences likely contributed to Gallup's less accurate vote estimate," their report states.
Gallup has transitioned back to a random-digit-dialing, "list-assisted" sampling frame, which it had used until 2011. "We have therefore made the corporate decision to change back" to random-digit dialing, Newport said.
One factor that Gallup says didn't lead their results to be inaccurate was the rise in cell-phone respondents. Gallup began conducting 40 percent of its interviews via cell phone in 2011, when it moved away from randomly dialing cell phones. They even increased their cell phone sampling to represent half their interviews last fall, which is greater than the percentage used by other pollsters.
Some of the other experiments found not guilty of causing inaccurate results include Gallup's use of a rolling sample, or "tracking design." That means that Gallup's daily results don't really represent separate polls. To wit: Gallup's most recent presidential approval tracking poll covers interviews conducted over the prior three nights. The next day's result will report the next rolling sample -- meaning it will include roughly two-thirds of interviews previously reported.
"We weight every night independently," Newport said. "One of the great advantages of our tracking program is that we can look at 30,000 interviews or 60,000 interviews" and look at various factors that could affect voters' opinions, Newport said.
Gallup also looked at the whether calling respondents -- after they are selected randomly, either from a list or by a computerized random dialer -- three times missed harder-to-reach voters, who may have been more Democratic. They conducted an experiment in which they called respondents five times, instead of three. "The initial results," Newport said, "show it did not make a difference," but more analysis is forthcoming.
Other Gallup experiments included identifying the pollster verbally and on caller-ID displays with the generic name "Selection Research" instead of "Gallup" and differences in the race and gender of interviewers. But they found that those factors did not significantly explain Gallup's inaccuracies last year.
Gallup's process is ongoing, and even if the changes they implement do address their 2012 issues, it's not clear that they can offset the changes that are affecting the industry at large. Survey costs are increasing as Americans move away from landline phones, and those reached are less likely to participate than they used to be than they were before. Newport said Gallup's response rate -- that is, the percentage of voters they attempt to contact who complete the survey -- was roughly 10 percent in 2012, a significant decline.
Still, Newport is optimistic that the review will lead to better results for his firm, whose name Traugott said "is synonymous with political polling all around the world."
"When the next presidential election rolls around," Newport said, "we think we'll certainly be in a position at the accurate end of the spectrum."
This article is from the archive of our partner The Wire.