One year of fact checking--an accounting
By Glenn Kessler,
It’s been one year since The Washington Post relaunched The Fact Checker column as a permanent feature, and so it seems an appropriate point to review and reflect on a year of fact-checking claims made by politicians. Most important, where did we go wrong and how can we improve?
Readers frequently ask: Do you rate more Republicans than Democrats? (Or vice versa). Which party gets the most Pinocchios? We had no idea until we sat down this week and did some calculations.
Let’s do the numbers!
When we last did this exercise, at the six-month point, it turned out that Democrats and Republicans were exactly tied in terms of the statements evaluated. But during the second six-month period, we evaluated more Republican statements than Democratic statements. This was largely because of the contest for the GOP presidential nomination, which accounted for about two-thirds of the Republican statements that came under scrutiny (especially when the candidates were trading charges about each other, not just about Democrats.)
All told, in the course of the year we evaluated 104 statements by Republicans and 101 statements by Democrats.
We also wrote at length about each GOP debate and reporter Josh Hicks evaluated claims GOP candidates made about their past. In retrospect, we can see why some readers complained in recent months that we have devoted too much attention to Republicans compared to Democrats. From a fact-checking perspective, it would have been nice to have had an equally fierce nomination fight on the Democratic side, with debates between candidates, but there is nothing we can do about that.
Nevertheless, no single person was evaluated more than President Obama (23 ratings). The White House sometimes disagreed strongly with our rulings.
Democrats fared slightly better than Republicans in terms of Pinocchios. They have an average of 2.32 Pinocchios per statement, compared to 2.49 Pinocchios for Republicans. As we noted at the six-month point, some of the GOP presidential contenders (We are looking at you, Rep. Bachmann) were largely responsible for inflating the Republican score.
The average Pinocchio rating of the political candidates is constantly updated by our Pinocchio tracker. Obama, with a vast White House staff to vet his statements, at the moment is basically tied for the lowest average with his nearest rival, former Massachusetts Gov. Mitt Romney. Romney also has a large campaign staff who help vet his statements, so it should not be a surprise that these two politicians have the best Pinocchio ratings.
Some readers have complained that the “average Pinocchio rating” is a bit absurd because it is completely dependent on a random selection of statements that we vet. This is true; it is inherently arbitrary. But we would also argue that over time one can get a broad sense of how accurate a politician is, and so that makes the average ranking is a rough but imperfect guide.
The hardest part about the job continues to be deciding how many Pinocchios need to be awarded. Since we do not use ½ Pinocchios, there are a bunch of 2’s that could have been 3’s — or vice versa. (The 1’s and 4’s are easier to spot, though readers sometimes vehemently disagree.) We admit the process is somewhat subjective, but the rating system also forces us to achieve a certain level of consistency.
Still, as an example of how subjective some of the ratings may be, it is interesting to note that two of our eight biggest Pinocchios of 2011 – Nancy Pelosi’s claim about senior citizen lunches and Joe Biden’s claim about rapes in Flint, Mich.—were rated “half true” by Politifact. Generally, the major fact-checking organizations come to more or less the same conclusion. But not always, which shows that there is always room for debate about our judgments.
For that reason, there are times when we do not award Pinocchios, but use the column to help explain a difficult issue or provide additional context to a politician’s statement. Some readers, however, have criticized us when we have not awarded Pinocchios. For instance, we did not reach a judgment on the GOP’s charges about Obama’s treatment of Israel. In that case, we laid out the facts and let readers come to their own conclusions. But many readers were not happy with that approach and wanted a more definitive rating.
Similarly, we also experimented with a “true but false” label, such as on the difficult question of Social Security’s contribution to the national debt. Some readers felt we flinched when we applied that same label to GOP’s claims about the number of small businesses that would be affected by a tax on millionaires. (Other fact checkers flatly labeled the claims as wrong.)
In a complaint that puzzled us, some readers wrote that we were consistently harder on female politicians than male politicians. These complaints especially came when we wrote about Democrats, such as Rep. Debbie Wasserman-Schultz, Rep. Nancy Pelosi or Sen. Barbara Boxer. We re-read every column that featured a female politician and concluded that the headlines included phrases that may have colored reader’s impressions, such as “ridiculous” and “absurd.” We tend to use such phrases for high Pinocchio ratings, men and women alike. But in the future perhaps it would be better to just stick with more neutral headlines.
We also regret referring to Margaret Sanger, the founder of Planned Parenthood, as a “racial pioneer.” We thought the full sentence in our column made it clear that we were trying to place her racial attitudes into the context of her age, but it still was a poor choice of words that some readers found objectionable.
With an unofficial quota of one column a day, there are bound to be some clunkers. Readers may have their own opinions about the dumbest or most wrong-headed column, but our nomination is this one: “Obama’s denial that Biden called tea party activists ‘terrorists’” We intended it as a guide for readers on how to tell the difference between verified fact and journalistic rumor, but its execution was poor and it was interpreted as an attack on Politico’s reporting. One of their columnists even wrote a widely quoted item titled “The End of Fact Checking.” Yikes!
We never intended to be journalism critics, and we will try to keep our focus on the politicians in this election season. We thank the many loyal readers of this column—and also our critics, who often write thoughtful responses that help keep us on our toes. Please keep those suggestions for columns—and improvement—coming in the new year.