Joshua Tucker: The following is a guest post from political scientists Francisco Cantú (University of Houston) and Marco A. Morales (Instituto Tecnológico Autónomo de México) as part of our continuing series of Monkey Cage Election Reports.
The runoff in the presidential election in Brazil will take place this weekend between the incumbent President Dilma Roussef (PT) and challenger Aécio Neves (PSDB). According to the conventional wisdom, the runoff is supposedly too close to call. Even the endorsement of Neves by the third-place candidate on the general election, Marina Silva (PSB), did not change that.
But in an all-too-familiar scenario, the accuracy of the polls has been questioned. Even the British weekly the Economist was quick to point to the difficulty of predicting the outcome of the general election in Brazil. In their view, the few (roughly 40) surveys published during the three months of the presidential campaign and the limited number of elections (six in total) since the return of democracy in 1989, make forecasting of Brazilian elections particularly hard.
In our view, with only four polling firms publishing their survey estimates during the campaign, we should not so much be concerned with ease of forecast but with the extent of potential biases in public poll estimates. Having fewer pollsters might raise the stakes for presenting skewed numbers, but it also provides fewer opportunities for each pollster to “check” their numbers against their peers’. But on the positive side, a healthy competition among pollsters might just be the necessary ingredient to foster good polling.
All of this becomes particularly relevant in the context of the “polling crises” that have been reported recently in various countries, including in other recent Latin American presidential elections such as Mexico in 2012, Honduras in 2013, or El Salvador, Costa Rica and Colombia in 2014.
This, then, begs the question: How did Brazilian pollsters perform in the first stage of the election? To answer that question, we deployed a statistical model that allows us to estimate the true vote intention throughout the general election, and use this information to calculate the systematic bias of each pollster. This is similar to what Simon Jackman used in his analysis of “house effects” for the 2004 Australian federal election, or what Drew Linzer used to forecast the Electoral College votes in the 2012 U.S. presidential election. In our case, we used all polls published in Brazil starting in early February of this year and ending with the last published surveys before the election. (Full details of the model can be found here.)
Brazilian electoral legislation requires pollsters to publish vote estimates for all candidates, as well as null and don’t-know votes. It also compels pollsters to publish a brief methodological summary and file it with the electoral authority. Having this information readily available allowed us to separate signal from noise and track true vote intention for each candidate throughout the campaign and its associated 95 percent credible interval. Our task is facilitated by the fact that we know the actual result of the general election.
Looking at these estimates, it would be hard to make the case that either the World Cup (June 12–July 13) or even the strident defeat of the Brazilian national team before Germany (July 8) had any noticeable effect on vote intent for Roussef. But we can say that the tragic death of PSB candidate Eduardo Campos and the entry of Marina Silva to substitute for him was a game changer in the election for a brief period.
Since we have a very good idea of how vote intent for each candidate moved throughout the campaign, we can also compute the systematic error with which each pollster estimated each candidate throughout the campaign. Results appear in the graph below, which shows the distributions of systematic error throughout the campaign that is independent from sampling error. Distributions appearing on the positive (right) side of the graph imply that a pollster tended to overestimate a candidate throughout the campaign, and distributions on the negative (left) side imply underestimation.
Now we can answer the initial question: How did Brazilian pollsters do in the general election campaign? In general terms, we see a mixed bag. Vote intent for Dilma Roussef seems to have been systematically overestimated, although some firms performed better than others. Curiously enough, despite the candidate change for the PSB, most pollsters seem to have estimated both Eduardo Campos and Marina Silva rather accurately, although with a very similar direction and magnitude of error. Perhaps most interesting is that most polling firms seem to have consistently underestimated Aécio Neves, who ended in second place on the general election.
Now that we know that no candidate got the necessary majority to win the election outright, we will have two very different moments that will be interesting to contrast: the general election, which was strictly multipartisan, and the runoff, which will be strictly bipartisan. It shall be interesting to see how pollsters performed when this next round is over.