Plain Text: 'Full of Holes'

Pundits and politicians, bloggers and businessmen spent much of Election Day confident in the knowledge that George W. Bush's tenure in the White House was over. They were reading and trusting the unofficial exit polls that showed a slim margin of victory for Sen. John Kerry. The official results, of course, told a different story.

The exit polls, produced by an organization called the National Election Pool (NEP), suggested all kinds of ultimately flawed results. For most of the day, they put Kerry ahead nationwide by a one- or two-point margin and with slim leads in Ohio, Florida and New Mexico--decisive swing states that he ended up losing. It also showed the senator competitive in states like Virginia, where he actually got whomped, and revealed a late-breaking, nonexistent surge of undecideds for Ralph Nader. The faulty figures appeared throughout Election Day on Web sites such as and on the sites of many bloggers, who hinted at a decisive Kerry victory. The data was then amplified by millions of readers who eagerly relayed those numbers to colleagues, friends and family.

By midday on Tuesday, there was a spreading certainty among the chattering classes that voters had booted the president from office. Pollsters were privately calling the race for Kerry and declining market numbers revealed Wall Street's skittishness about the apparent change in administration. As the actual counting of the votes began, Republicans were panicky, Democrats energized. Early on election night on CNN, an obviously elated Ted Kennedy spoke with unusual confidence about a Kerry administration.

He should have recalled some recent history. This is the third black eye in a row for the unified exit-polling effort by six major news organizations: ABC, CBS, NBC, CNN, Fox and the Associated Press. Their previous consortium, the Voters News Service (VNS), humiliated itself and its media partners in 2000 when its numbers first suggested that Al Gore had won Florida, then George Bush, then that the race that was too close to call. In the 2002 midterm elections, the VNS failed to produce any useable results at all. But the news organizations need those polls to guide their live coverage and to help project the winners. So they pulled the plug on VNS and instead started a new organization, the NEP, retaining two veteran pollsters to conduct the operations.

The NEP was transparent about its methods. Before the election, it announced it would send its part-time workers to 1,480 precincts around the country chosen to reflect the nation's demographics. It also planned to collect results from another 3,000 precincts after the polls closed, and make thousands of calls before the election to voters in 13 states, to try to capture a representation of the early vote and the absentee vote.

Before the election, the NEP pleaded with its media partners to keep the exit polls secret during the election. "They are only estimates and should not be used in the reporting of any poll results on air, in print or on the web," went one NEP missive. This turned out to be naive. Keeping exit polls secret in today's wired world is like trying to stop rain from hitting the ground.

But there were larger problems than the leaks. Four years ago, it was the analysts who goofed, drawing faulty conclusions in an election that was too close to call. This time around, it seems that the data itself was bad. "We looked across every state and it looked like a Kerry sweep [of the swing states]," said Michael Maslansky of Luntz Research. His boss, GOP pollster Frank Luntz, sent out an e-mail to clients at 4 p.m. ET on Election Day extrapolating a huge Kerry win from the exit polls. "These guys were working on this for four years to get the methodology down. They are smart guys. You expect them to get it right," says Maslansky.

They didn't. The most obvious mistake seems to have been an overly large sampling of women voters, particularly early in the day. In the exit polling NEP released at 1 p.m. ET on the day of the election, of the 5,000 voters who had been interviewed, 59 percent were women. Three hours later, of the 8,349 voters who were interviewed, 58 percent were women. In both of these statistical snapshots, Kerry was winning the election, according to NEP. The organization knew its numbers looked sketchy. It reportedly held a midday conference call with its media partners to alert them to the problem. Still, University of Virginia political science professor Larry Sabato says the NEP "ought to give its money back to the news networks."

Sabato also says this: "I used to believe it was possible to structure exit polls in a way to make them accurate. I am now starting to wonder if it is even possible." In other words, exit polling--like polling itself--may simply be an unreliable science in today's world. Perhaps we are now too busy or impatient to talk to a pollster over the phone, or to stop to answer questions after waiting in long lines to vote. Early voting and absentee voting further complicates the challenge of accurate predictions. "Exit polls are full of holes," says Joan Konner of the Columbia University Graduate School of Journalism, who co-wrote an internal report for CNN about the 2000 debacle. "Nobody should ever take these things seriously."

But don't feel too sorry for the media organizations that paid for the bad data, or for the left-wingers whose hopes were lifted by the exit polls and then crushed by the real results. The real tragedy is that journalists and historians must now analyze the election without a clear window into what the voters actually thought. And that is truly scary. Exit polling has traditionally played another important function, helping to reveal the real intentions of the voters. As University of Michigan professor Michael Traugott says, they provide "a public protection against the false claim of a mandate."

In his acceptance speech, the president laid claim to such a mandate. "America has spoken, and I'm humbled by the trust and confidence of my fellow Americans," he said. The exit polls, were they bankable, might tell another story: of an America apprehensive about changing leaders in the middle of the war against terror but still deeply suspicious about the direction the country is heading in. But all we can truly know about 2004 is the bottom line: 51 percent of voters checked a box for Bush, 48 percent checked one for Kerry.