The Slippery Art of Polling

If quantity fostered quality, this year's polls asking "McCain or Obama?" would be the best in history: polls have proliferated so crazily that Mark Blumenthal, who analyzes them at pollster.com, writes in a typical post, "Another day, another 37 new statewide polls." The profusion isn't surprising. It reflects the tightness of the race, the increased emphasis on state rather than national polls as the Electoral College looms large, and a race so fluid that pollsters are bound and determined to capture every shift. What is surprising is that, after a primary season in which polls got several big races wrong (Clinton. New Hampshire. Enough said), pollsters are breaking ranks on some key aspects of methodology and admitting that, this year, old truths about polling are in tatters.

No controversy looms larger than that about "likely" voters. Polls that count them, rather than registered voters, are usually more accurate, but maybe not this year. Pollsters determine who is likely to cast a ballot by asking questions such as whether they voted in 2004, are following the campaign and plan to vote. Respondents get a point for each "right" answer. The pollster then takes a percentage of the sample equal to the percentage of adults who voted (60 percent in '04) and includes only that percentage of scorers. The bottom 40 percent don't count as "likely." This can produce inaccurate results in many ways. For one thing, if 70 percent rather than 60 percent of voters cast a ballot this time, pollsters may be ignoring 14 percent of their sample. For another, because the millions of newly registered voters score zero on the "did you vote last time?" question, they might not count as likely. But if they do turn out on Nov. 4, polls are failing to count a group that could determine the outcome in several states. "The problem with the likely-voter model is that if you have a transformative election, you will have a different turnout," says Nate Silver, who uses polls as input for his forecasting model at fivethirtyeight.com. "My hunch is that there will be more and different voters this time," putting polls on shakier ground.

Cell-phone use might also skew polls. In standard polling methodology, a computer program selects landline numbers at random; laws prohibit random-digit dialing to cell phones. Pollsters have to call cell phones manually, which is time-consuming and expensive. Cell-only voters—now 13 percent of the population, and mostly under 30—are therefore absent from polls. Does that skew results? When ABC News called cell numbers manually, the impact was "negligible," it reported this month: a gain of 0.7 points for Obama and a loss of 0.8 for McCain among likely voters. Those are all smaller than the poll's margin of error. Gallup and the Pew Research Center find that Obama gains no more than 2 to 3 points when they call cell phones.

Why isn't the effect larger? Because of one of the key arts of polling: demographic weighting. If the reported results are, say, 60-40 in favor of McCain, it doesn't mean that 60 percent said they favored the Republican while 40 percent favored Obama. Instead, the pollster counts how many respondents came from each gender, age group, ethnicity, region, even education level. Then the pollster weights the answers. If random dialing yielded 350 men out of 1,000 respondents, and there are 480 men per 1,000 registered voters, then each man's McCain-Obama choice is up-weighted by a factor of 1.37 (480/350). That's how pollsters adjust for any undercount of young cell-only voters: they up-weight the responses of young voters reached on landlines. This works unless young cell-only voters are politically different from young landline users, as a new study from Pew suggests. Of the under-30 cell-only population, 62 percent were Democrats and 28 percent Republicans, Pew finds, and they preferred Obama by 35 points, almost triple the number of under-30 landline users.

Few things infuriate critics more than polls that have "too many" respondents of a particular party. The Gallup organization regularly catches flak for supposedly overpolling Republicans, for instance. But few pollsters adjust for party self-identification as they do for gender, age and other fixed categories. The reason is that party ID is surprisingly fluid. In a 2005 study, Larry Hugick of Princeton Survey Research Associates International (who polls for NEWSWEEK) found that 18 percent of voters reported a different party ID in October 2004 than they had the month before. It may seem that a poll with fewer self-identified Democrats than the percentage of total voters in the last election who were Democrats, or with less than the percentage of total voters who are registered as Democrats, has undersampled them. In fact, however, the party that respondents say they belong to may reflect their candidate preference. In this case, Dems moving to McCain change their party ID as well and say they're Republicans. Down-weighting GOP responses because a poll has "too many" Republicans would therefore underestimate McCain's support.

Oh, and something else about polls. Just because they don't agree with the Election Day tally doesn't mean they were incorrect at the time they were done. If the election were held a week later, then news events, shifting mood and voters' most recent exposure to a candidate and his message could easily yield a different result than on Nov. 4.

© 2008