Can we trust the polls?
That’s a real question these days. Pollsters have been trying — and not always successfully — to adapt their methodologies to keep up with cultural and technological change. With caller ID, many people no longer answer phone calls from numbers they don’t recognize, or refuse to answer questions from strangers if they do. As they abandon landline phones, young voters become more expensive to poll on their mobile devices. Given campaigns’ promises to mobilize reluctant voters, it’s difficult to predict which kinds of people will actually vote in traditionally low-turnout primary elections — which is essential if pollsters are to project winners and losers.
Those difficulties have been on display in some high-profile failures to predict how citizens will vote. For instance, few pollsters predicted that the United Kingdom would so decisively vote for Brexit. Other surprises include Bernie Sanders’s upset of Hillary Clinton in the Michigan primary this year; Ted Cruz beating Donald Trump in this year’s Iowa caucus; and the outcome of the 2015 U.K. general election. Some in the prediction business have wondered if we’re moving toward a “post-polling world.”
Would prediction markets — in which individuals bet on the outcome of particular votes — be any better?
Many observers (including Justin Wolfers, John Stossel and Steven Rattner) have argued that such markets are indeed more accurate at forecasting electoral outcomes. Academics have studied the accuracy of betting on political outcomes since the 1980s, so there’s rich research literature to draw on.
Those who believe in forecasting markets argue like this: Polls reflect the daily maelstrom of political news, as voters are buffeted by competing considerations, with their opinions fluctuating based on that day’s events. Betting markets, on the other hand, force traders to soberly parse the both the structural and the emergent political factors — lest they lose money. Just as Samuel Johnson declared the threat of hanging “concentrates the mind wonderfully,” so, this argument goes, does risking one’s assets — forcing those who bet to be systematic and discriminating. Undisciplined traders are ruined, and the prescient are enriched. Over time, the invisible hand should find the most insight. What’s more, bettors can read the polls, so presumably they take those into account.
But the data suggest there’s not much difference between polling and betting markets.
Consider the figure below, which shows national polling and betting data for the last five competitive major party primaries. (I am using primaries because political scientists know they are particularly hard to predict, since voters can’t take their cues from the party itself.) I took the polling data from Huffington Post’s poll aggregator. I took the betting data from overseas bookmakers for 2016 such as BetFair, William Hill, BWin, Sky Bet, totaling 18 bookmakers in all; for earlier cycles, the data come from Intrade.
The left column shows polling averages for the eventual winner (in black) and their competitors (in blue). The right side of the graph plots the betting data. To make the two quantities comparable, I converted the betting data into a probability of victory based on the candidate’s share price.
If one approach predicts better than the other, it should show clear and early separation between the winner and the runners up. That’s the telltale sign of a leading indicator.
But we find that only for the 2012 Republican primary. In the polling, a series of candidates — Herman Cain, Newt Gingrich, Rick Santorum — briefly outpolled the eventual nominee, Mitt Romney, before fading. In the prediction markets, we see a brief flirtation with Santorum, but that comes to an end early. Romney, the eventual nominee, reliably led betting markets for more than a year before the New Hampshire primary.
That’s not so in the four other elections.
Consider the 2016 Republican primary. Jeb Bush led all betting markets until October 2015 — by which time he’d been at single digits in the national polls for more than three months. When Bush faded in the prediction markets, who took his place? Not Donald Trump, who had led every national poll for months, but Marco Rubio.
Bettors didn’t overwhelmingly put their money on Trump until January 2016. And when Ted Cruz won the Iowa caucus, Trump tumbled more than 30 points on the betting markets, only bouncing back after winning New Hampshire the next week.
Throughout this entire period, the polls had Trump in the lead. In other words, the polls were more accurate than the prediction markets.
The figure below summarizes which national primary polls and betting markets predicted the outcome more accurately. I look at the 2016 Republican and Democratic primaries; the 2008 Democratic and Republican Iowa caucus and New Hampshire primaries; and the 2012 Republican primaries. (Because President Obama faced no serious opposition in 2012, I didn’t include that year’s Democratic contest.)
Each black line represents a single election and shows how well the polls or betting markets did at predicting the eventual outcome. Higher numbers are better. In statistics terminology, they’re reporting the r-squared estimates for a group of linear models.
The red line in each graph summarizes the various contests. The value for polls is 0.57; for the betting markets it’s 0.61. That’s not a big difference. And in the period 100-200 days before the first primary, the polls do significantly better than the markets.
the informed opinion that Trump was politically ephemeral. Maybe they thought that the strong elite GOP opposition would halt Trump’s rise. Such theories were lucrative in previous elections for those who shorted the protest candidates’ fleeting popularity. But they were wrong; the polls were right.
One blogger, Daniel Bier, argues that the U.S. betting markets failed because they were too heavily regulated and allowed insufficient funds to be wagered to allow for weighting the competing accounts of the race — that is, the payoff wasn’t enough to weed out obtuse traders. He suggested that U.S. political prediction markets were therefore limited because bets were placed by a small number of traders reading the same stories and tweets, participating in the same conversations and adopting the same outlook.
But that’s false. In both figures above, the data are drawn from both small, regulated U.S. markets (like Intrade, now defunct) and large, liquid foreign bookmakers. Foreign bookmakers and domestic markets’ predictions were pretty closely aligned, with both prices correlated at greater than 0.9.
Despite the well-established difficulty of polling of the U.S. electorate, primary polls apparently deliver more information than bettors. The traders might wish to pay more attention to polls in future cycle. After all, it’s their money on the line.
Thomas Wood is an assistant professor of political science at Ohio State University who studies voter behavior and public opinion. Follow him on Twitter @thomasjwood.