There’s an idea in polling called the Bradley effect, which posits that polls misjudged Tom Bradley’s 1982 loss in California’s governor’s race because poll respondents were unwilling to admit that they planned to vote against the first black gubernatorial candidate in the state’s history. The idea, in other words, is that social pressure can shift what people will tell pollsters.
About a month before Donald Trump threw his hat into the presidential ring in 2015, Pew Research unveiled an interesting analysis of its polls that measured something similar. Pew compared responses given to live-caller pollsters with those from online surveys. In other words, it compared poll results given when a respondent had to talk to a real person with responses given in Web-based surveys.
They found some differences. People on the Web were more likely to give strongly unfavorable ratings of public officials and were less likely to say that they were satisfied with their family lives. But that’s not all.
“Questions about societal discrimination against several different groups also produced large differences,” Pew wrote, “with telephone respondents more apt than Web respondents to say that gays and lesbians, Hispanics and blacks face a lot of discrimination.” In other words, politically incorrect opinions, if you will, were more prominent online, when the poll respondent didn’t have to express that opinion to a real person on the other end of the phone.
During the presidential primaries, there was a gap in views of Trump between those taking polls online and those speaking to a live pollster. Trump was the politically incorrect candidate, if you will, and there was some speculation that this drove the split. When the general election rolled around, Trump adviser Kellyanne Conway said a similar split was keeping Trump’s poll numbers down — but his poll numbers from online surveys were no higher than his live-caller numbers.
But that split is back.
Charles Franklin, director of the Marquette Law School Poll, pointed out on Twitter and later, in a blog post, that Trump’s approval ratings in live-caller polls tended to be lower than Internet-based polls or Interactive Voice Response (IVR) polls, which run phone respondents through automated prompts. With that in mind, the fact that CNN’s new poll, conducted by SRSS, had him far lower than Rasmussen’s survey isn’t a surprise. CNN uses live callers. Rasmussen uses IVR.
Using data from HuffPost Pollster, we can look at Franklin’s analysis in more detail. Below, for example, are all of the results from approval polling since Jan. 20, 2017, broken out by the type of poll.
It’s busy, but what you want to notice is that the red-colored polls — those not using live callers — are generally higher than the blue-colored ones. Trump’s approval rating, which has ranged from a bit over 30 to the low 50s early on, has been higher in Internet-based polls than in live-caller ones.
Let’s add trend lines, making it easier to see the split. As Franklin noted, IVR polls have the highest Trump approval, followed by Internet-based surveys and then, at the bottom, live-caller polls.
It’s important to note that Rasmussen’s polls have consistently been higher than other pollsters and that they contribute most of the poll results to the IVR-based set. They also survey only likely voters, which is a population that can skew more Republican. It’s hard to tell, in other words, whether IVR polls are more favorable to Trump because of methodology or because of the pollster.
More interesting, though, is what happens when we do the same split looking at the approval ratings offered by respondents from each partisan group.
Republicans tend to view Trump positively.
Independents, less so.
Notice that among independents and Republicans, the blue dots are pretty evenly scattered among the red dots, suggesting little effect from the mode of the survey (that is, not much difference between live-caller and Internet-based polls).
If we add trend lines to the Republican chart, you can see that the two types of polls have been generally about the same over Trump’s presidency.
And then there are the Democrats. Here are approval rating results among Democratic respondents.
The blue dots are consistently lower than the red dots. Meaning that live-caller polls consistently give Trump lower approval ratings than online surveys.
Adding trend lines makes that very clear.
Democrats are a substantial chunk of the overall respondent pool, of course, so it’s safe to assume that the gulf above is a primary driver for the difference in approval ratings Trump sees based on the mode of the interview.
Or, put in English: Trump’s approval in live-caller polls is lower, it seems, because Democrats approve less of his job performance when talking to a live person than when answering the question online.
There is certainly an expectation for Democrats that they will not show support for President Trump. He’s deeply unpopular among Democrats regardless of the type of poll at issue. But it does seem possible — not confirmed, but possible — that there’s a social effect at play: that Democrats are less willing to tell a real person that they approve of Trump, just as some Americans in Pew’s research were less willing to tell a live person that they didn’t think gay men and lesbians faced a lot of discrimination.
This doesn’t mean that there is a secret pool of Democrats who plan to vote Republican in November or for Trump’s possible reelection bid in two years’ time. It does mean, though, that the type of survey should help inform interpretation of poll results. And that Democrats may be wary of telling other people that they actually sort of approve of Trump’s efforts in the White House.