Politifact Bias (or not) [Defending the Truth-O-Meter]

The Blog "Politifact Bias" seems to be a pretty popular blog for conservatives who want to confirm their poorly-justified belief that the neutral fact checking site known as Politifact is actually left-biased (at least according to Google Search Results). I have yet to look at this blog directly since they rarely publish original posts. However, I thought I may as well check out their "PolitiFact 2011: A review" post to get an idea of what they considered to be significant evidence of the diagnosed liberal bias of Politifact in 2011.¹ As far as I can tell, this blog merely collects criticisms of Politifact they find around the internet. In my experience, these criticisms can be extremely poor. They listed "a rundown of the issues that should keep discerning readers from trusting PolitiFact:"

1)  PolitiFact persistently ignores the effects of selection bias.  It simply isn't plausible that editors who are very probably predominantly liberal will choose stories of interest on a neutral basis without some systematic check on ideological bias.  PolitiFact, for example, continues to publish candidate report cards as though selection bias has no effect on the report card data.

Basically, the linked article⁴ is a great example for a subject I was planning on writing about, entitled "The Wrong Ways To Detect Bias." One of the methods I was planning on talking about was exactly what these people did. They just compiled a list of articles on Politifact over the course of a year and came to the conclusion that, since the worst ratings tend to favor republicans more than democrats, they must be biased in their selection in favor of the left. The problem with this is that their conclusion simply does not follow. Their are other possible reasons for this kind of result that have nothing specifically to do with Politifact being biased.

1. To give them the most credit, it is possible there may be some liberal selection bias at work here, but not in the way they think. Politifact often gets ideas for statements to check from their readers. It is possible most of their readers are liberal, which souds quite plausible given the fact that conservatives seem so ready to accuse Politifact of bias. This is no fault of Politifact, nor does it show any bias on their part. The fault lies squarely with Republicans who chose not to submit statements to Politifact and instead waste their time complaining about perceived bias (a vicious cycle). It should also be noted that the definition of Selection Bias is "A type of bias caused by choosing non-random data for statistical analysis." Well duh! The article itself mentions that Politifact Bill Adair noted "We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." That right there should show that there is selection bias, though not liberal selection bias. However, the article intends to show that the bias is toward the left, instead of toward important statements. If the article was intended to merely accuse Politifact of selection bias, they would basically be attacking a straw man since the main goal of Politifact is not to tally true and false statements to figure out which candidates are more honest (although this does happen occasionally). It should also be noted the selection of Politifact articles used in this calculation are very much non-random. As the article mentioned, the time period selected was when Democrats were in power. This could have easily affected the results, as I will get into later.


2. Could it be that Republicans were telling falsehoods more often during that time period? The article does mention this. "One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site. However, there is no evidence offered by PolitiFact that this is their calculus in decision-making." Yet they don't look for evidence that their explanation is the better one. To show the fallacy in this kind of thinking, let me present an analogy: Let's say i have two dogs that roam free in my house all day, Rex and Dex. I come home one day to find my couch clawed to pieces. I cannot find evidence that Rex did it, so I assume Dex did it (even though I found no evidence specifically pointing to Dex). This is exactly the same fallacy they employ. They don't specifically have evidence of a Republican tendency towards falsehoods so they assume it must be a Politifact tendency towards liberal bias. Like I noted earlier, the article did not randomly chose articles. Could it be that, since the Republican Party was out of power, they had a greater tendency towards falsehoods? Could it be that the ratings success of Fox News created a podium for more Republican falsehoods? Could it be that, with the rise of the right-wing Tea Party, extreme views have tended to produce more falsehoods? Could it be that the possible rise of anti-intellectualism in the Republican Party has made it more likely to back falsehoods? Could it be that the Republican Party itself is more prone to being dishonest? Could the Republican party be more ideological and less prone to checking facts that could challenge their ideology? Could it be that the Republican tendency to call neutral fact-checking sites "biased" creates a tendency to play fast and loose with the truth? These are all possibilities (although I am not necessarily claiming any of them to be true). So did the article control for or check any of these? NOPE. They could have checked other neutral fact-checking sites to see if they came to similar conclusions as Politifact. They could have compared these results with articles from before Obama was elected. They just dismissed these hypotheses for no better reason than they would have for dismissing their own hypothesis.


3. It could be that the significance of the truer statements was much higher than that of the false statements. Not all statements are equal. Chris Christie's statement about the city of Newark's graduation rate has far less of an impact on party image than the Democratic claims about Republicans trying to end Medicare. An overall look at a large number of graded statements can give you a decent look at a candidate's honesty, but you need to actually check the statements to see how significant they are. If Politifact is giving Democrats more high-profile false's then it doesn't matter that there are fewer. The primary focus of Politifact once again is to read the about ratings they give these statements. So a handful of high profile falsehoods for Democrats can look much worse than a bunch of inconsequential falsehoods for Republicans.

It should be pretty clear that merely tallying Politifact ratings doesn't give you anything remotely conclusive about the theoretical bias of a site.

2)  PolitiFact continues to publish obviously non-objective stories without employing the journalistic custom of using labels like "commentary," "opinion" or even "news analysis."  Readers are implicitly led to believe that stories like an editorial "Lie of the Year" selection are objective news stories.

Any evidence for this? A link or two maybe? If they wrote an article about this, it would be useful at this point. I may search for an article in their archives and come back to this later. As it stands, this is an unsubstantiated statement. I will, however, agree that the "Lie of The Year" title is a bad title since there is a difference between people who lie (know their statements are false) and people who are careless with the truth (don't necessarily know their statements are false). However, Politifact does have some leverage here since the 2009, 2010, and 2011 Lies of the Year have been tackled by Politifact, as well as other Fact Checking sites, so often, it is hard to imagine these politicians do not know their claims have been debunked. It is also good to note that Politifact lets readers know the "Lie of The Year" is just a consensus opinion of their editors and reporters, not objective news. So readers are actually not led to believe the Lie of The Year is an objective news story.

3)  PolitiFact continues to routinely apply its principles of analysis unevenly, as with its interpretation of job creation claims (are the claims assumed to refer to gross job creation or net job creation?).

The link provided goes to a re-post of this exact same blog entry. I'm interested to see what they are talking about. If I find evidence I will post my critique.³

4)  PolitiFact has yet to shake its penchant for snark.  Snark has no place in objective reporting (see #2 above).  Unfortunately, PolitiFact treats it like a selling point instead of a weakness, and PolitiFact's intentional use of it has apparently influenced Annenberg Fact Check to follow suit.

I'm not 100% sure what he means by "Snark"² However, I will assume he means "Use of sarcasm or malice in speech." I'm not sure I have ever seen sarcasm. But I fail to see the problem here. Politifact is in the business of spreading awareness of "Fact Checking." It helps keep readers interested and coming back for more (although it's nowhere near as common as in Glenn Kessler's articles). Overall, it has no negative effects on the actual research. In fact, it can help get readers fired up about falsehoods and dishonesty. Other than personal taste, I see no problem with this. In fact, any case of "snark" I have ever seen has been quite subtle, meaning I cannot see it honestly turning away any readers that care more about reality than ideology.

It seems as though none of these "issues" should "keep discerning readers from trusting Politifact" so long as they care about quality fact-checking to help ensure they don't subscribe to false beliefs. Only one of these "issues" even had a linked article, and the content of that linked article came nowhere near demonstrating bias at Politifact. I will empathize that showing the existence of bias is no easy task (usually). But if you are having THAT much trouble exposing bias, maybe you should wonder if the bias really exists at all...


¹ - Update 1-10-12: The writers of "Politifact Bias" have informed me that this post was meant to be just a summary, and was not meant to give significant evidence of bias. However, this does not excuse the fact that they did list "the issues that should keep discerning readers from trusting PolitiFact," without providing evidence for a few of them. This does not change the rest of this post because all this post does is check to see if these "issues" significantly exist at all.

² - Update 1-10-12: The writers of "Politifact Bias" have informed me that, by snark, they mean that Politifact is conveying their own biases. When I asked for an example, they provided me with a quote from a July 2011 politifact article talking about Newt Gingrich:"With the 2012 election in their sights, Republican candidates spend most of their time trying to prove that President Barack Obama and the Democrats will make the economy worse." However this is a terrible example given that this is not an uncontroversial fact. There is no hint of the reporter's feelings, biases, or prejudices. Nor is there any hint of unnecessary partisan adjectives. It sounds as if this example fails to substantiate their claim that Politifact is guilty of "snark." 

Update 1-10-12: I have edited this post to enhance readability. No significant changes were made to the content.
 
³ Update 1-10-12: A writer of "Politifact Bias" has informed me that the link was to a keyword search on one of his blogs (seems a bit unnecessarily distracting and cryptic). He verified a certain post did a good job underscoring his complaint. In this post he mostly just lists each case and states whether it was helped or harmed by the inclusion/exclusion of "gross jobs." The problem is he mostly fails to investigate context. There are some statements where it is appropriate to look at net jobs, such as a statement that looks at "jobs added" as a whole. Conversely, there are statements where it is appropriate to ignore lost jobs. When a politician states the stimulus did not create one job, it doesn't matter how many jobs were already lost as a result of the recession, just so long as at least one job was directly created by the stimulus. Since the mechanism for lost jobs and the mechanisms for gained jobs are different, it is more than appropriate to ignore what has nothing to do with the mechanism under question. For example, If I went to a village under attack by militants and pulled a single child to safety, thereby saving it's life, should I not be credited with saving that kid's life just because more than one life was lost in that village from the attack? Any reasonable person would find this absurd. Granted he attempted an investigation into the context of choosing based on the word "created." I will give him some credit for that, but he did not investigate whether or not Politifact was consistent when the mechanism that "created" jobs was the same as the mechanism that caused jobs to be lost. He just simply focused on the word "created." As the stimulus example shows, his list definitely ignores important context. It should also be noted that Politifact has actually made the context used in their decision making well known (see last link).

⁴ - Update 1-16-12: I have also completed a comprehensive critique of this article as well.