How to use fact checkers? Ignore the conclusions

There’s a new study out today claiming that “PolitiFact rates Republicans as the less credible party” during Barack Obama’s second term. John Sides quickly shreds that to pieces:

Politifact isn’t randomly sampling the statements of Republicans and Democrats.  They’re just examining statements they consider particularly visible, influential, or controversial.  The data are consistent with any number of interpretations and so we can’t say all that much about the truthfulness of political parties, about any biases of Politifact, etc.

Exactly correct. There’s simply no way of knowing whether PolitiFact has included more Republican “lies” because Republicans lie more often; because PolitiFact has a bias for choosing false Republican statements (or perhaps true Democratic statements) for evaluation; or even for biases in how statements are evaluated. (If there are biases, they could be partisan ones, or they could be in-party/out-party ones, or any number of other possibilities). Given all that, there’s just no way, as John says, to draw any conclusions about either the parties or PolitiFact from these data.

To back up a bit: I’ve been very critical of fact checkers, but I still do believe that they are a positive development. My general sense is that the strongest work they do is in digging up actual factual information. PolitiFact, The Post’s Glenn Kessler and others tend to do an excellent job of bringing in good information, usually supplying solid citations and maintaining a healthy respect for the complications of “fact” analysis.

They have larger problems (as media scholar Brendan Nyhan has pointed out) with their ratings. There, they fall back on judgment calls, often with little obvious objective ways to distinguish how many “Pinocchios” something is worth or whether something is “half true” or “pants on fire,” or even which claims should properly be considered claims of fact. That’s not because they’re bad at their jobs; it’s usually because the job, so defined, really isn’t very doable.

On top of that, for all their flaws, the ratings — often thrown up on screens in attack ads — probably do provide healthy incentives for campaigns to avoid whoppers.

For the rest of us, as hard as it is to resist temptation, the best thing to do is probably to entirely ignore the conclusions and be grateful just for the research because it’s futile to wring information about party mendacity from the ratings.