![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Polls! Polls! Polls!
Aug. 7th, 2008 07:08 pmIn 2004, some people started putting up websites on which they tried to aggregate state-by-state polls to produce estimates of how the battle for electoral votes was shaping up in the presidential election. Some of those have started doing it again. One of the most popular ones was electoral-vote.com; there was also RealClearPolitics and Election Projection and some others. The political sympathies of these people were across the spectrum (electoral-vote's Votemaster seems to be a centrist Democrat; RCP and Election Projection are conservative), and that sometimes affected which polls they trusted and consequently their results; sampling bias is a hard thing to deal with here. But they all actually did an OK job of tracking how things were shaping up.
In the end, things were so close that the situation was very confusing, but the last week or so of polls actually tracked Bush's narrow reelection win pretty well--it was only the infamous leaked early exit polls that were strangely off in Kerry's direction. (Some people took this as a sign that there had been some sort of massive nationwide vote-rigging, but I was convinced by the Mystery Pollster's various arguments that, whether or not that happened, the early exits weren't good evidence for it.)
Anyway, a really popular new one is FiveThirtyEight.com on the liberal Democratic side. The guy who runs it, Nate Silver, is apparently better known for his sabermetric baseball analyses. What he does is a more elaborate version of what a few sites attempted in 2004, basically a Monte Carlo simulation. He takes state polls, fuzzes them out with probability distributions calculated according to a fairly complicated model (weighting polls by how old they are and how reliable a track record the pollster has, in addition to the supposed margin of error from pure sample size), puts in some other contributions calculated from national polls, and then does thousands of simulation runs to get a distribution of electoral votes.
He says he's actually trying to predict the November election, and as a result he puts a lot of extra slop into the probability distributions to reflect what he doesn't know. Which is fine, but reading his site I often get the impression that the evolution of his numbers depends more on the frequent changes he makes to his complex model than anything in reality. I think I'd prefer a simpler model that remained more constant over time.
An interesting recent development was the return of Sam Wang at Princeton, who was one of the slightly-lesser-known people doing this in 2004. Wang made a recent post on his blog criticizing FiveThirtyEight.com, pointing out, among other things, that if what you want is a probability distribution calculated from a bunch of state Gaussians, you really don't need to do a Monte Carlo simulation; you can just calculate values of the aggregate distribution directly and get more accurate results with fewer cycles. It's a good point, though it's also true that Silver's approach gives you a lot of fun (if questionably accurate) additional details about the probability of dozens of different weird things happening. Wang's got his own model, for which he makes no claims of real prediction--he just calls it an "if the election were held today" aggregate--so it isn't far off FiveThirtyEight's in overall result but has a much tighter spread.
Wang has reason to be skeptical of models with a lot of ad-hoc assumptions. Late in the 2004 campaign, he started to post pairs of electoral cartograms--one calculated just from recent polls, and another that incorporated his speculative assumption, based on observation of previous elections, that undecideds would break preferentially for Kerry. In the end, the tweaked map was wrong and the one that just aggregated polls was pretty much dead on. He learned from the mistake: here he reprints a pugnacious e-mail chalking it up to wishful thinking and his remarkably polite response.
In the end, things were so close that the situation was very confusing, but the last week or so of polls actually tracked Bush's narrow reelection win pretty well--it was only the infamous leaked early exit polls that were strangely off in Kerry's direction. (Some people took this as a sign that there had been some sort of massive nationwide vote-rigging, but I was convinced by the Mystery Pollster's various arguments that, whether or not that happened, the early exits weren't good evidence for it.)
Anyway, a really popular new one is FiveThirtyEight.com on the liberal Democratic side. The guy who runs it, Nate Silver, is apparently better known for his sabermetric baseball analyses. What he does is a more elaborate version of what a few sites attempted in 2004, basically a Monte Carlo simulation. He takes state polls, fuzzes them out with probability distributions calculated according to a fairly complicated model (weighting polls by how old they are and how reliable a track record the pollster has, in addition to the supposed margin of error from pure sample size), puts in some other contributions calculated from national polls, and then does thousands of simulation runs to get a distribution of electoral votes.
He says he's actually trying to predict the November election, and as a result he puts a lot of extra slop into the probability distributions to reflect what he doesn't know. Which is fine, but reading his site I often get the impression that the evolution of his numbers depends more on the frequent changes he makes to his complex model than anything in reality. I think I'd prefer a simpler model that remained more constant over time.
An interesting recent development was the return of Sam Wang at Princeton, who was one of the slightly-lesser-known people doing this in 2004. Wang made a recent post on his blog criticizing FiveThirtyEight.com, pointing out, among other things, that if what you want is a probability distribution calculated from a bunch of state Gaussians, you really don't need to do a Monte Carlo simulation; you can just calculate values of the aggregate distribution directly and get more accurate results with fewer cycles. It's a good point, though it's also true that Silver's approach gives you a lot of fun (if questionably accurate) additional details about the probability of dozens of different weird things happening. Wang's got his own model, for which he makes no claims of real prediction--he just calls it an "if the election were held today" aggregate--so it isn't far off FiveThirtyEight's in overall result but has a much tighter spread.
Wang has reason to be skeptical of models with a lot of ad-hoc assumptions. Late in the 2004 campaign, he started to post pairs of electoral cartograms--one calculated just from recent polls, and another that incorporated his speculative assumption, based on observation of previous elections, that undecideds would break preferentially for Kerry. In the end, the tweaked map was wrong and the one that just aggregated polls was pretty much dead on. He learned from the mistake: here he reprints a pugnacious e-mail chalking it up to wishful thinking and his remarkably polite response.