The Washington PostDemocracy Dies in Darkness

The shadow stalking Democrats’ improving numbers: Can polls be trusted?

Mail-in ballots for the general election in West Chester, Pa., on Oct. 23, 2020. (Matt Slocum/AP)

Over the past two months, FiveThirtyEight’s average of congressional ballot polling has shifted from a two-point Republican advantage to a half-point Democratic one. Its average of approval polling for President Biden has gone from Biden being underwater by 17 points (that is, his “disapprove” percentage is 17 points higher than his “approve”) to a more modest 11 points. Speaking a bit more figuratively, Democrats generally have moved from a position of panicked hyperventilation to one of cautious breath-holding.

But then there’s that shadow that follows the party around like a rain cloud: What if the polls are wrong? Or, really: What if the polls are wrong again?

Sign up for How To Read This Chart, a weekly data newsletter from Philip Bump

When Republicans overperformed polls in 2014, not many people noticed. Republicans held an advantage in generic-ballot polling (poll questions that pit an unnamed Republican against an unnamed Democrat) and things simply went worse than expected. Two years prior, it was Barack Obama who beat the polls so, you know. You win some, you lose some.

Then came 2016. Hillary Clinton had leads in swing state polls and in polling averages. Models gave her strong odds to win. Democrats were confident. But many people didn’t notice how the models and the polls shifted in the last few days of the election. Then came Election Day and, well, you don’t need me to explain what happened.

In 2018, things worked out for Democrats again, and it seemed that maybe the ghost had been exorcised. Democrats were expected to do well, and then Democrats did well.

Then came 2020. Joe Biden’s lead was wider than Clinton’s and observers (ahem) noted that even a Clinton-level miss in state polling wouldn’t keep him out of the White House. Then there was a near-Clinton-level miss, and Biden eked his way into the presidency.

So we come to 2022. Will it be like 2018, when Trump wasn’t on the ballot and the polls were on the money — Democrats squeaking into retaining a House majority? Or will it be like 2014 and 2016 and 2020 — worse than expected?

That question of Trump being on the ballot is how I’ve considered things for some time now. Then I saw a tweet from the Economist’s G. Elliott Morris (who has a newly published book about polling) that made an important point: While the national generic-ballot poll hit the mark in 2018, state-level polls were still off the mark.

I decided to put this to the test. I pulled state-level polling averages for Senate contests from FiveThirtyEight and compared them with actual vote outcomes. The pattern is pretty clear: Senate polling averages consistently underweight the performance of Republicans.

How to read this chart: At top are beeswarm plots positioning each Senate contest according to how far its predicted results (the polling average) were from the actual results. States in which Democrats beat the polls are in blue; states where Republicans did are in red. States that flipped from 2016 to 2020 are in bold.

At bottom is an admittedly complicated comparison of the polling averages to the results. The polling average margin is shown from top to bottom, with a bigger Democratic advantage at the top and a bigger Republican one at bottom. From left to right, the actual margin, with Democratic wins to the left of the centerline and Republican ones to the right. The key is the diagonal: states to the right of it saw Republicans overperform the average and states to the left saw Democrats overperform.

This is dependent on the FiveThirtyEight average, but since it’s one of the tools people use to evaluate how things will go, I feel comfortable using it as the metric. (If you’re curious, I’m using the average that includes consideration of non-polling factors, like fundraising.) And what we see is a pattern of those averages of conducted polls underestimating how Republicans would fare.

In a tightly divided Senate, the possibility that polls will again be off, even by a little, means a change in control of the chamber, as Morris noted on Twitter.

“If you believe Senate polls today have no national bias toward the Democrats, then they are ~80% to hold the majority (based only on polls, not fundamentals),” he wrote. “If you think polls will be as biased as in 2020, they are slight underdogs.”

In other words, just enough races will go the other way to give Republicans a slight advantage. It’s easy to see where that might happen. In Pennsylvania, FiveThirtyEight has Democrat John Fetterman up by about five points. In 2020, it figured Biden would win the state by about five points. He won by just over 1 point. In 2018, Democrats overperformed in the Senate race there by nearly two points, however. And in 2016, they underperformed by a slightly bigger margin.

(Let me note that this is not meant to disparage FiveThirtyEight! It just has the misfortune of having objective data about polling that it presents in a usable format, which I appreciate.)

Of course, there’s another factor at play: the passage of time. We are still about 70 days out from the election during which a lot can happen. This month, I looked at how the national generic-ballot had changed in the last four midterm election cycles over the last 90 days. In two, Democrats improved and, in two, Republicans did. Importantly, during the two years in which the national average was distant from the actual results, the shift was against the Democrats.

This is a small sample size! And I admit that I don’t have a good answer for the question of what we might expect. It’s a weird year with weird dynamics (a hyperpresent, unpopular former president) and enormous political shifts (the overturning of Roe v. Wade) that are having an effect.

But I can say this: that shadow that lingers around Democratic poll numbers is, indeed, something that might justifiably prompt a few chills down the spine.