In an age of anti-poll sentiment, what can we learn from France? On Monday, WorldViews spoke with Claire Durand, president of the World Association for Public Opinion Research and a professor at the University of Montreal, to ask what the French polls got right — and what lessons there might be for pollsters who are accused of missing Trump and Brexit.
WorldViews: Overall, how well do you think the French polling companies did?
Claire Durand: They did very, very well. They are within the margin of error everywhere. Their prediction for Macron is on the spot. Le Pen? It is within the margin of error, but she has been a bit overestimated. [Leftist Jean-Luc] Mélenchon has been underestimated, which is a surprise, though it's not necessarily an underestimation, I think. What happened most probably is that many people who thought they were going to vote for [Socialist Benoît] Hamon went to Mélenchon at the last minute.
Their [political platforms] were very close, and as people thought that perhaps Mélenchon could make it to the second round, they may have left Hamon and gone to Mélenchon.
When we look at the methodology of these polls, are there certain things that went right or went wrong?
It is always difficult with French polls because they use reports of previous elections to help estimate. They are not the only ones who do that. It's a bit like in the United States when people use partisanship. It tends to reduce variance . . . and people get accused of plots like “herding,” as in the last British general election. [Herding is when pollsters make decisions that cause published estimates to vary less than expected.] That didn't happen this time, though.
What is extremely interesting in France is that there is a law where all the pollsters have to file all the information about their polls with the French polling commission. I'm on the AAPOR [American Association for Public Opinion Research] committee for the U.S. election and if we had that, we'd be very, very happy!
[French polls] ask people how they voted in previous elections, like the 2012 elections and the regional elections in 2015. I looked at two pollsters — IFOP and OpinionWay — because they give all the information . . . and really it was almost perfect. This means that when they use “report of previous election” to adjust their estimates, it doesn't change anything. In fact, what you have is almost exactly what they got anyway, since they used quotas for certain demographics. What you see is that whether they weight or adjust or do anything else, it does not change the overall numbers. It's rather interesting. It was not like that in 2002, I can tell you.
That was the last time the National Front got through to the second round. Was that a big surprise?
2002 was for France what 1948 was in the United States or 1992 in Britain: It was a very big miss [for political pollsters].
At that time, there were three leading candidates. Lionel Jospin [of the center-left Socialists], Jacques Chirac [of the center-right Rally for the Republic] and Jean-Marie Le Pen, father of Marine. Everybody was absolutely sure that the two candidates that would make it to the second round were Jospin and Chirac. And what happened? Le Pen made it to the second round.
Afterward we were informed by some people that at least two pollsters had Le Pen ahead of Jospin, but they changed the numbers! It was a bomb. People had no confidence in the polls, so there were some changes and the polling commission started to check more. They looked at one pollster's numbers and said the estimation does not correspond to the data file. And the pollster said, “Yes, I thought my numbers were not good and, you know, this would not be good for the credibility of polls.”
In 2016 there was a new change in the law and now everything is on the web so everyone can consult it. Interestingly, now there is no more underestimation for the extreme right.
That's something that people talk about a lot. I think the average person now thinks that the extreme right is always underestimated in polls. That wasn't the case here?
No. In fact, I went back and looked at the 2012 elections and things were already better by then. In France, people now say, “Voting for the extreme right is not shameful anymore.”
From what we know, did Le Pen get a bump in support after the attacks last week?
We don't have that data, because it happened so late. We don't think it was the case.
Overall, it seemed that the only thing the polls missed was turnout, which they thought might be lower than 2012. Why was that?
Estimating those who will or will not vote is an old problem for survey methods. Even if you ask people whether they voted [after the election], the estimate is actually quite a lot higher than reality. My estimate was that the turnout would be high as there was other research in political science that shows that the more candidates you have and the closer the election, the more people go vote. They think their vote can change something.
There have been other elections around the world — for example, in the U.S. or Britain — where there has been a big backlash to polls, rightly or wrongly. Is there anything that these polling companies could learn from the French pollsters?
Yes. Explain what the margin of error means! For me, it's the lesson of the U.S. and it's the lesson of Brexit. What I've seen in France is that all along, everybody spoke about the margin of error [and that it] means it could be either/or.
They did not use that much of the probability thing that you see in the United States. I'm absolutely against that: I think that one reason Clinton lost in the U.S. election was that aggregators published figures showing she had high probability of winning based on published polls. When you tell people it is a high probability it will rain, they think it is sure that it will rain. We didn't see that so much in France. What we can learn is: Explain clearly what the margin of error means. In referendums like Brexit or a close election in the U.S., it is extremely important.
We're now heading up to the second round. Most of the polls I've seen suggest Macron has a comfortable lead. Is there any reason to doubt the second-round polls?
No. I've seen how they do it, it's almost mathematical. They ask people how they voted in the first round. They adjust for that and only for that and then they run their estimation process. This usually gives an exact — absolutely exact! — estimate. They've never missed the second-round vote. In fact, they usually have it perfectly.
This transcript has been edited for clarity and length
More on WorldViews