Quora Question: Who Is the More Credible Presidential Pollster, Nate Silver or Sam Wang?

11_04_earlyvote_01
Voters cast their ballots during early voting at the Beatties Ford Library in Charlotte, North Carolina, on October 20. Chris Keane/Reuters

Quora Questions are part of a partnership between Newsweek and Quora, through which we'll be posting relevant and interesting answers from Quora contributors throughout the week. Read more about the partnership here.

Answer from Paul Mainwood, academic:

Both Silver and Wang use credible methods, both based on the most historically predictive data for Presidential elections: state-level polling.

In this, they are similar to Drew Linzer's Votamatic (hosted by Daily Kos this year), and Upshot's aggregation for The New York Times. These models all use credible methods and should be distinguished from irritating variations on a theme of "This Amazing Political Scientist has a model based on the ECONOMY that predicted the last zillion Presidential ELECTIONS and they say My Candidate will WIN."

However, even these four differ on some important choices of how to aggregate these polls. Right now, one week out from the election, the difference is particularly stark between Wang and Silver, so it's a good time to look at the differences.

I'll start with Wang's approach, because it's the cleanest (and also open-source, which Silver's is not).

  • Take the quality state polls as they come out, weight the latest ones more highly, older ones less so, and use their margins of error to construct a Bayesian probability distribution of belief on who is likely to win the electoral votes in that state
  • Calculate all possible variations of state results with their associated probabilities, to arrive at a national distribution of electoral votes for each candidate: count the probabilities up until one candidate gains 270 EVs
  • Project this distribution out into the future with some variation on a random walk (Wang uses a pure random walk as well as a more sophisticated "Bayesian Drift" approach).

Silver performs a similar process, but with a few variations.

  • 538's model takes national polls into account. Wang's approach does not (except as a guide to the variance in the random walk in step 3).
  • 538 has a weighting and correction system for pollsters that weights lower for those with uncertain track records, and also makes explicit adjustments for past Republican/Democrat bias. (To their credit, 538 are transparent about this, and report both the original numbers and adjustments.) Wang doesn't correct anyone.
  • 538 weights current polling higher than past results (i.e., their prediction comes out more "twitchy" to recent news.)
  • 538 assumes that there may be systemic bias (i.e., correlated bias) across the reads from all state polls, and also that there will be correlations of poll movement into the future. Wang assumes only that there will be correlations in future movement.

The last two are the dominant effects and are where the majority of the delta between the forecasts comes from. I cobbled this graph together to show just how different the two approaches can end up.

Black line = PEC (Wang): on 99% Clinton win today (1 November)

Blue line = 538 (Silver): on 76% Clinton win today (1 November)

The other two aggregators fall within the range defined by these two:

Upshot/NYT: 88% Clinton

Votamatic/Daily Kos: 96% Clinton

[Update: there's a new guy in town, an open-sourced implementation of Linzer's model: results and all R code is freely available for download here. Unsurprisingly, their numbers track Votamatic closely.]

As you can see, PEC is almost always above 538 (that is, more certain). This is mostly down to 538's allowance for systemic bias across the state polls. PEC's projection is also much more stable (this is down mainly to its higher responsiveness to recent news).

So who is more credible?

They both have approximately equal track records across the period they've both been aggregating. They got the overall horse-race right every time since 2008, and both got almost all the states right too. But this isn't surprising, those elections were very easy to predict (bar a few states that both models correctly rated as a toss-up: e.g., Florida).

In the past, Wang's overall predictions were more certain (i.e., closer to 0 or 1), and settled down to their final values sooner, just like this year. Silver's kept on bouncing around until the last moment, again, just like this year.

Sam Wang has been doing this for longer. He first deployed his model in 2004 (the model got the result right, but he added some additional assumptions about late-breaking voters that made him predict the wrong result—he learned from this not to make manual adjustments!)

Nate Silver is the more famous and has the greater reputation to lose. He has been repeatedly criticized by other poll aggregators (including Wang) for lumping in extra volatility and uncertainty in a quest for clicks, an accusation he rejects.

The difficulty is that we have not had a really harsh test of presidential elections to separate out the methodology of the models. The closest we've had were the mid-term elections of 2014, where there was a lot of uncertainty around and only sketchy polling to go on. The best performer here was neither PEC nor 538…it was Drew Linzer's Votamatic.

My advice would be read a bit more about the methodology and make your own choice. My own position is more towards Votamatic and PEC than 538. And since the betting markets seem to be driven more by 538's probabilities, the last few days have offered me a whole bunch of opportunities to put my money where my mouth is.

Never bet more than you can afford to lose, but if you share my opinion, then there are some good odds to be had right now.

Who is more credible as an election forecaster – Nate Silver or Princeton's Sam Wang? originally appeared on Quora - the knowledge sharing network where compelling questions are answered by people with unique insights. You can follow Quora on Twitter, Facebook, and Google+.

More questions:

Quora Question: Who Is the More Credible Presidential Pollster, Nate Silver or Sam Wang? | U.S.