Political fact-checkers, like our own Glenn Kessler or the Tampa Bay Times's PolitiFact, are in an unenviable position. There so many politicians and pundit spreading so many misinterpretations, misleading characterizations, and outright falsehoods that it'd be impossible for the checkers to catch them all. They'd be forgiven for occasionally wondering if their project was actually useful.
Turns out it is. Two political scientists — Brendan Nyhan at Dartmouth and Jason Reifler at the University of Exeter — have conducted a field experiment (summary here) that shows that state legislators really are less likely to mislead the public when they know fact-checkers are watching.
Here's how it worked. In fall 2012, Nyhan and Reifler looked at nine states that had their own PolitiFact affiliates. They randomly selected a group of those states' legislators and sent them notes informing them of the study and reminding them of the presence of PolitiFact affiliates and the negative consequences to their careers that could result from lying. Members of a second, placebo group were sent letters that informed them of the study but did not mention PolitiFact or any negative consequences of lying. A third group got no letters at all.
The idea was to see if the group that got the reminders about PolitiFact acted any differently from the other groups. They did. Those receiving the warning letter were 55 percent less likely to get a negative rating from their state's PolitiFact affiliate during the 2012 election season. Their odds of having the accuracy of statements they had made questioned, either by PolitFact or other media, fell by 75 percent.
This probably substantially underestimates the power of fact-checkers, since, by necessity, they lump in politicians who read the warning carefully with those who just ignored it.. "We can only estimate the effect of being assigned to receive the treatment letter," the authors write. "It is unlikely that every state legislator to whom we sent the treatment letter received it and read it carefully. If the negative consequences of inaccurate statements were salient and accessible to all elites, the potential effects on their behavior would likely be even larger." What's more, fact-checkers of necessity can't evaluate every statement a candidate makes, which makes the fact that there was a still an effect on fact-check grades, even when candidates know their odds of getting evaluated are low, all the more striking.
This isn't a slam dunk. It's entirely possible that the treatment group responded by discerning what kinds of false statements fact-checkers tend to challenge, and limiting their falsehoods to other areas. But the study makes clear that there's at least some response to the presence of fact-checkers, even if that kind of premeditated lying muddies the waters a bit.
And the results suggest that higher volume fact-checking operations, which can evaluate a greater share of candidates' statements, would be more effective still. The main takeaway to get from this is that we need two, three, many Glenn Kesslers!