A public argument has erupted between two leading forecasters of the Senate battle — the internationally famous Nate Silver, and the little known Sam Wang, a professor of neuroscience at Princeton who dabbles in election forecasting as something of a hobby. The battle is being treated as a bit of a sideshow — as a duel of dweebs.
But the underlying dispute between them is actually quite important, and has long term ramifications for how we think about polling and elections.
Today Silver kicked things up a notch with a broadside aimed at Wang, and in a conversation with me today, Wang responded.
Wang gives Democrats good odds of holding the Senate, while Silver still marginally favors Republicans.
In a nutshell, the dispute turns on the difference between their two models. Wang’s model is a “polls only” model that bases predictions on a median of all available public polling data. Silver’s model is premised on the idea that polls alone aren’t enough, and adds in a number of “state fundamentals” to his model, including the generic ballot, Congressional approval ratings, fundraising totals, the background and ideology of the candidates, and so forth.
That model is wrong — not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically — that is, by looking at how accurate polls or polling averages have been in the past — Wang makes several assumptions about how polls behave that don’t check out against the data.
Silver points to several instances in which Wang’s model got it wrong in 2010, and explains why he layers in his fundamentals this way:
One reason is that you sometimes have no alternative; the occasional Senate race gets literally no polling. Or it gets very limited polling…Another reason to look beyond polls is to prevent abrupt shifts in the forecast….the state fundamentals estimate is based on a series of non-polling indicators that have historically shown some predictive power in Senate races…
Asked to respond, Wang conceded that polls aren’t necessarily perfect, but he argued that the layering in of fundamentals also adds an element of uncertainty that risks being arbitrary and biasing the outcome. Here’s Wang’s (slightly edited) response:
Political science tells us that campaigns are about driving elections towards or away from a natural outcome. When modelers make models, they are trying to predict a ‘natural’ endpoint. There is always the possibility of biasing your actual calculation. Adding in assumptions like fundamentals could be right and could be wrong, but what they represent is basically a hypothesis about where the race ought to be. Making a model like that is a good political science experiment, but we should be clear that we are adding assumptions on top of the polls. This introduces an additional level of uncertainty. Anytime you add another component to a model, you are always adding uncertainty. You have to be cautious about whether you’re adding more signal, or whether you’re adding more noise. It’s conservative procedure to be cautious about adding assumptions. I think we should keep the assumptions and the polling separate.
Presidential models have been tested for decades. In cases where two candidates are running nationwide, political science has a lot of experience modeling which variables affect the outcome. Senate races are harder because there are local factors that can surprise analysts. Polls have the ability to capture those. And they have the ability to capture the fundamentals, too — they are baked into the polls. Because polls are a way to ask people who they are going to vote for. This is common in the sciences — there’s a direct way to find out something and an indirect way to find out something. The direct way is usually better.
Silver insists Wang’s model has failed in the past. But Wang responds that his has actually proven more accurate than Silver’s in Senate races, citing his own record of calling all the 2012 Senate races accurately:
Poll medians had a perfect record in 2012, whereas FiveThirtyEight did not. As an empirical statement of fact, poll medians did better than polls-plus-fundamentals. As a population, professional pollsters are very good at what they do, and what I’m trying to do at my site is give a relatively clean read on the wisdom of that crowd.
Soon enough, we’ll find out whose model is superior. Of course, it’s very possible that the two will converge as more and more polling data becomes available. But ultimately the outcome very well may shed light on the best way to read polls — and how much weight to give them.