Do Polls Still Work?

May I be the first to say ‘Mr. President’?” inquired Bob Shrum, rather too memorably, of John Kerry on Election Day 2004, as early exit polls showed his candidate cruising to victory. It turned out Shrum’s faith was misplaced—so badly that the episode ranks alongside “Dewey Defeats Truman” in the annals of Great Political Embarrassments.

After faulty reading of exit polls caused the Election Day 2000 debacle, this latest confusion suggested a deep futility in even trying to gauge public opinion, before or after an election.

With this month’s elections shaping up to be another nail-biter, the question “Do polls still work?” is increasingly practical. Pollsters no longer seem like wizards. It’s no secret that today they must contend with skyrocketing refusal rates and obstacles like “cell-phone-only households.” So partisans feel free to criticize them—especially when their side is losing.

The last election season saw oddly technical attacks: the liberal group MoveOn.org attacked the Gallup organization in The New York Times over its “likely voter methodology.” And startlingly vehement ones: Republicans picketed the Minneapolis Star Tribune, pounding on windows and hollering at employees, for insufficiently weighting conservatives in its survey samples.

Pollsters themselves freaked out. Contrary to their partisan caricature, legitimate practitioners of the calling are a docile and introverted breed consumed with little beyond their own narrow subspecialties. If you don’t know one, a good example is William Hurt’s character in Altered States—the rumpled academic so engrossed in his research that he fails to notice when his marathon stints in a sensory-deprivation tank transform him into a rampaging man-beast.

The polling community reached for its heavy artillery: the strongly worded journal article. Last year, Public Opinion Quarterly, the premier journal in the field, devoted a special issue to an autopsy of the 2004 preelection polls, and proved the naysayers wrong. The studies found that the polls had been among the most accurate in half a century, and concluded that blame for the mess lay elsewhere: reporters—always suckers for drama—had treated statistically insignificant day-to-day shifts as major stories rather than predictable aberrations, thereby heightening the up-one-day, down-the-next whiplash effect that tends to undermine confidence in any poll. No wonder the public was angry! For all the criticism, one author observed, most polls “were among the best recorded in the contemporary period.” Gallup’s last poll gave Bush a 2-point lead; he won by 2.5 percent. It was, as the issue’s coeditor, Lawrence Jacobs, put it, “a kick to the groin of conventional myths about polling.”

And what about Bob Shrum and the exit polls? As a series of postelection studies took pains to note, the self-styled experts drastically misread the raw exit-poll data that leaked on Election Day. Instead of seizing on the early numbers and declaring a winner, experts now tell us, bloggers and the press should have waited for all the data to be properly weighted and balanced.

The National Election Pool—the outfit that runs the exit polls—has made one useful concession to reality and set out to stop leaks from racing across the Web, hitting upon an ingeniously low-tech solution to a high-tech problem. This year the pollsters will be placed in a “quarantine room” and stripped of cell phones, pagers, Internet access, and other means of outside communication. They won’t be allowed back into the electronic real world until many hours later.

If all goes according to plan—and if the pollsters locked in the tank don’t turn into rampaging man-beasts—the results we hear later that evening will be as accurate as they should be.