The Weekly Standard Attacks Fact Checkers (Part 3)

 

This is part 3 of a 3-part series entitled "The Weekly Standard Attacks Fact Checkers." Part 1 can be read here. Part 2 can be read here. 

SEEN IT BEFORE

Hemingway later decides to mention a familiar article.  

While there’s been little examination of the broader phenomenon of media fact checking, the University of Minnesota Humphrey School of Public Affairs recently took a close look at PolitiFact. Here’s what they found:

"A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts. In total, 74 of the 98 statements by political figures judged “false” or “pants on fire” over the last 13 months were given to Republicans, or 76 percent, compared to just 22 statements for Democrats (22 percent)."

You can believe that Republicans lie more than three times as often as Democrats. Or you can believe that, at a minimum, PolitiFact is engaging in a great deal of selection bias, to say nothing of pushing tendentious arguments of its own.

At the very least, this is a nice little false dichotomy, I have already examined this "study" in a previous post¹ and found it to be shoddy work. The writer did not implement practically any controls in the study. He also dismissed or ignored alternate explanations for reasons that should have given him equal justification to dismiss his own hypothesized explanation. I'd venture to guess this article would be thrown out in a second if submitted to a legitimate peer-reviewed journal.

On August 17, Kessler wrote an item supporting President Obama’s denial at a town hall in Iowa that Vice President Joe Biden had called Tea Party activists “terrorists” in a meeting with congressional Democrats. In the process, Kessler had singled out Politico for breaking the story. ...

After supplying a rudimentary summary of what happened, Kessler reached a conclusion that is at once unsure of itself and sharply judgmental. “Frankly, we are dubious that Biden actually said this. And if he did, he was simply echoing what another speaker said, in a private conversation, as opposed to making a public statement.” In response, Smith unloaded on Kessler. “Either [Biden] said it, or he didn’t. That’s the fact to check here. The way to check it is to report it out, not to attack the people who did report it out and label their reporting ‘dubious’ based on nothing more than instinct and the questionable and utterly self-interested word of politicians and their staffers.” Provoked by Kessler, Politico took the unusual step of actually detailing how the Biden story was nailed down. Politico maintains that Biden’s remarks were confirmed by five different sources in the room with Biden, and that they were in contact with the vice president’s office for hours before the story ran. Biden’s office had ample opportunity to answer the reporters’ account before it ran and didn’t dispute it. ...

But instead of looking at these facts, it appears Glenn Kessler engaged in what his colleague Greg Sargent referred to as all “the usual he-said-she-said crap that often mars political reporting”​—​but with the extra dollop of sanctimony that comes from writing under the “pseudo-scientific banner” of “The Fact Checker.”

To a certain degree, they do have a point. However, Kessler has also pointed out the article was never meant as an attack on Politico, just "as a guide for readers on how to tell the difference between verified fact and journalistic rumor." Such a distinction is necessary for readers to make informed decisions over how much credibility they should give a story. He reported that there were conflicting accounts of what happened, which should be no surprise given the unreliability of eyewitness testimony. Politico's sources could have easily misinterpreted what he said, which is all to plausible. If I said "dealing with you is like dealing with a child," does that mean I actually called you a "child"? A few people overhearing my conversation may have thought so, even if that was not my intent. One could argue I implicitly called you a child, but it should be very clear how easily they could be constructing a straw man. Joe Biden agreeing with the idea that "negotiating with Tea Partiers is reminiscent of negotiations with terrorists" is a far cry from him explicitly calling them "terrorists." The difference is crucial to any adult who feels they have matured past the age where they may accuse their friends of "saying an icky word!" These are the kinds of issues brought up by Kessler, and an examination of the available evidence submitted by politico, as well as Kessler (in his original article), shows one should be skeptical of any headline that reads “Biden: Tea Stands for ‘Terrorist.’” It should also be noted that Kessler even admitted he did a poor job of writing the article.

AND HERE COMES THE BIAS

Well that is the last of the examples given by Hemingway to help support their barrage of claims. After a few more paragraphs of unsubstantiated claims, Hemingway finished with this little doozie:

So with 2012 just around the corner, brace yourself for a fact-checking deluge. Just remember: The fact checker is less often a referee than a fan with a rooting interest in the outcome.

So if it wasn't completely clear before, Hemingway is claiming these fact checkers are biased. And given the fact that all instances mentioned in this article are against Republicans, its clear he is accusing them of being left-biased. Any evidence provided in this article has been shown to be shoddy at best. In addition, one would expect that, if these people REALLY have a vested interest in liberal success, they wouldn't post articles that really damage strong liberal talking points.

• One popular talking point against Obama, during this election season, is that he has failed to keep his promises. Obama tried to counter this by saying he has kept 60% of his promises, to which Politifact gave him a "false". The real number is 30%. He has kept more of his promises than people give him credit for, and only broken about 10%, so there is a grain of truth to this. However, since he doubled the actual number, he rated a false. Remember how Hemingway complained when Rand Paul was given a false for quadrupling is numbers? I wonder why there wasn't equal outrage from Hemingway over a relatively less-deserved "false" rating for president Obama? Is it because Obama isn't conservative and thus does not fit into his liberal bias paradigm?
• Politifact, likely the most target of these sites for liberal bias claims often cites conservative/libertarian think tanks such as Cato Institute, Heritage Foundation, and American Enterprise Institute (btw, also notice the many popular Democratic talking points that are debunked in these articles).
• The 2011 "Lie of The Year" was the incredibly popular campaign claim from Democrats that "Republicans plan on eliminating medicare." This claim already helped propel at least one Democrat to victory in a heavily Republican district. Seeing just how valuable this message is to Democrats in the coming election, it seems highly unlikely the message would be debunked by anyone remotely interested in Democratic success, let alone be given the less-than-coveted "Lie of The Year" award.

And of course, these are only a few examples that show just how absurd the Hemingway liberal bias claim really is. Could these fact checking sites be producing these just to try and make themselves look non-partisan? Outside the subjective choice of "Lie of The Year," that would seem to be quite the claim to substantiate, and is itself an example of the popular ad hoc fallacy, "evidence against bias is evidence for bias." However, without this kind of conspiratorial response, it seems the best explanation is that these fact checkers are in fact not fans "with a rooting interest in the outcome."

SO WHAT'S THE BOTTOM LINE?

This all isn't to say that fact checkers are not prone to subjectivity, or mistakes. They are human after all. At the very least, this should underscore the need for readers to never suspend a reasonable amount of skepticism, even when reading fact checks. However, unlike everyday political pundits, fact checkers still attempt to remain accurate and objective. If politicians and media pundits did the same, perhaps fact checkers would no longer be so sorely needed.

Overall, the central thesis of Hemingway's article appears to be that, if a politician says his/her claim is opinion, it should be free from the scrutiny of fact checkers. However, the practice of fact checking is essentially a form of Skepticism. Fact checkers assume the role of non-partisan skeptics who concern themselves with reality first and foremost. A common practice of skeptics is to highlight common fallacies so others can spot them. This underlies the fact that a person can come to accept a false belief based on fallacies, as well as facts. For example, Politifact pointed out that Rand Paul failed to provide context with his claim, which is essentially a form of cherry-picking. It is also the job of the skeptic to help others identify shoddy or less-than-optimal evidence, which should inspire doubts in skeptical minded individuals. Glen Kessler did just this when he pointed out the difference between a public statement and private rumor. In general, if a statement contains fallacies, can be verified true or false, or leaves out important context, there is no reason why skeptics should not be able to critique said statement. Should the breadth of a false or fallacious statement mean it should be left alone? Should skeptics take off their critical thinking caps when reading the Opinion page? Any good Skeptic knows that all statements are up for scrutiny because statements that do not reflect reality deserve no place within public discourse. As practicing skeptics, political fact checkers should be expected to hold to this very same principle, exposing bunk in political discussion, regardless of where it occurs.




¹ - Update 1-16-12: I have also completed a comprehensive critique of this article as well.

The Weekly Standard Attacks Fact Checkers (Part 2)

This is part 2 of a 3-part series entitled "The Weekly Standard Attacks Fact Checkers." Part 1 can be read here.

DEFENDING THE TRUTH-O-METER

The AP's Fact Check was not the only target of this article. What is a good attack on the practice of Fact Checking without a good shot at Politifact?

Here’s a not-atypical case study. On November 7, 2010, newly elected Senator Rand Paul appeared on ABC’s This Week with Christiane Amanpour. One of the topics of discussion was pay for federal workers. “The average federal employee makes $120,000 a year,” Paul said. “The average private employee makes $60,000 a year.” Given that the news these days often boils down to debates over byzantine policy details, Paul’s statement is about as close to an empirically verifiable fact as you’re likely to hear a politician utter. And the numbers are reasonably clear. According to the latest data from the Bureau of Economic Analysis​—​yes, that’s a government agency​—​federal workers earned average pay and benefits of $123,049 in 2009 while private workers made on average $61,051 in total compensation...

(section removed due to lack of relevance)

Not only is what Senator Paul said about federal pay verifiably true, his simple recitation of the most basic facts of the matter doesn’t even begin to illustrate the extent of the problem. Yet PolitiFact rated Senator Paul’s statement “false.” According to PolitiFact’s editors, because Paul did not explicitly say the figures he was citing include pay and benefits, he was being misleading. The average reader would assume he was only talking about salary. “BEA found that federal civilian employees earned $81,258 in salary, compared to $50,464 for private-sector workers. That cuts the federal pay advantage almost exactly in half, to nearly $31,000,” writes PolitiFact. So the average federal employee makes a mere $31,000 more a year in salary than the average private sector worker​—​but also gets a benefits package worth four times what the average private sector worker gets. PolitiFact further muddies the waters by suggesting that the discrepancy between public and private sector averages isn’t an apples-to-apples comparison. Again, Andrew Biggs, the former Social Security Administration deputy commissioner for policy, and Jason Richwine of the Center for Data Analysis, writing in these pages (“Yes, They’re Overpaid: The Truth About Federal Workers’ Compensation,” February 14, 2011), observed that the most favorable studies of federal worker compensation “controlling for age, education, experience, race, gender, marital status, immigration status, state of residence, and so on” still find federal workers are overpaid by as much as 22 percent.

At the very least, this criticism underscores the principle that most falsehoods have a small grain of truth. However, Politifact has to give Rand Paul's statement a rating based on the expected interpretation of a reasonable person. Since he included both pay and benefits, without specifying so, this alone can arguably justify a bad rating as a viewer could easily overestimate the amount of money a federal worker makes. In addition, Rand Paul makes it seem as though federal pay is DOUBLE private sector pay. Politifact notes that, once the comparison is apples-to-apples, that number is closer to $7000, or a little more than 10% greater than comparable private sector jobs. How does Hemingway respond? He retorts that "the most favorable studies" (if you read the article, you will notice that all these studies are either by conservative think tanks or cherry picked government jobs) actually put that number as low as 22%. However, 22% is still a far cry from 100%. So, even by Hemingway's standards, the actual differences are much less than what Rand Paul cites. How does quadrupling the difference in pay not warrant a major downgrade in rating?

THE AP IS NOT OFF THE HOOK YET

Hemingway then returns to his attacks on the AP's fact checking operations. During the nomination of Justice Elena Kagan, the AP ran a Fact Check article dealing with claims that she was anti-military. Hemingway decided the AP was fact checking opinion and not actual facts.

Again, here are the facts: Kagan was a dean at a law school that had banned ROTC over what she referred to as the military’s “repugnant” ban on openly gay service. This was, not surprisingly, an issue raised when she was nominated for her current position on the Supreme Court. The AP’s own fact check even noted that she filed a legal brief in support of colleges that wanted to uphold their policies restricting military recruiters on campus, though she opted not to join the lawsuit. Whether the fact that Kagan valued making a statement about gay rights over supporting the vital national security effort of military recruitment amounts to being “antimilitary” is quite obviously a matter of opinion, as is the charge that she’s an “ivory tower peacenik.”

Actually the prefix "anti" literally means "one that is opposed." So it stands to reason that "anti-military" means "one that is opposed to the military." This is a factual claim, not an opinion. It is also demonstrably false. The AP points out: "In the heat of the recruitment debate, and after, Kagan praised military service as "the noblest of all professions" and a "socially valuable career path" that should be open to all. "I know how much my security and freedom and indeed everything else I value depend on all of you," she told West Point cadets." She even sent out an email defending "the school's earlier decision to set aside restrictions on the armed forces and to allow - not ban - military recruitment. (emphasis mine)." Maybe Hemingway can explain  how supporting the military can possibly be consistent with being anti-military.

It was actually Newt Gingrich who made the claim on "Fox News Sunday." Fact checkers have to evaluate statements based on how a reasonable person would interpret them. There should be no argument that someone who heard Newt Gingrich's statement would think Elena Kagan is actually against the military, based on his statement alone. Furthermore,  "Whether the fact that Kagan valued making a statement about gay rights over supporting the vital national security effort of military recruitment amounts to being “antimilitary” is quite obviously a matter of opinion" is not a standard I'd venture to think Hemingway would follow consistently. If Democrats were to say that Republicans are utterly opposed to fiscal responsibility because they favor avoiding new taxes over balancing the budget, I doubt Hemingway would treat such a claim as merely opinion.

Revealingly, the inflammatory phrase “ivory tower peacenik” was never actually used by Kagan’s critics​—​it was from the AP headline and the first sentence of its fact check: “Elena Kagan is no ivory-tower peacenik.” Here the AP pulled off a seriously impressive feat of yellow journalism. By caricaturing the tone of the actual criticisms, the AP set up a straw man for its “fact check” to knock down before the reader even got past the headline.

It should be noted that the AP never claimed Republicans called Kagan an "ivory-tower peacenik." In order for this to actually be a straw man, the AP would have had to criticize this characterization of Justice Kagan. In fact, nowhere in the article did the AP do that. The irony here is that Hemingway himself essentially criticized a straw man instead of what the AP actually wrote. Hemingway should also know "ivory-tower Peacenik" is merely a catchy title. Seeing as how nowhere in this article does Hemingway actually diagnose fact checkers as liars, as his title suggests, the question arises over whether or not he is, by his own standards, guilty of yellow journalism. Since I doubt he would stoop to this level of hypocrisy, I doubt he could fairly justify the claim the AP is itself guilty of yellow journalism.

The Weekly Standard's attack on the AP Fact Checking operation continued:

At the most basic level, the media’s new “fact checkers” remain obdurately unwilling to let opinions simply be opinions. Earlier this year the AP fact checked a column by former GOP presidential candidate Tim Pawlenty in which the former Minnesota governor asserted that “Obamacare is unconstitutional.” Contra Pawlenty, the AP intoned, “Obama’s health care overhaul might be unconstitutional in Pawlenty’s opinion, but it is not in fact unless the Supreme Court says so.” The AP aligns itself here with the myth of judicial supremacy, namely the mistaken idea that the Supreme Court has a monopoly on deciding what is and is not constitutional. But aside from this amateur-hour excursion into legal theory, the AP betrays a more basic problem of reading comprehension: Pawlenty’s USA Today column appeared in a section of the newspaper clearly labeled OPINION in large, bold letters.

Hemingway has completely confused me. The AP clearly admitted that Pawlenty was merely expressing his opinion. So why is Hemingway accusing them of faulty reading comprehension? Did Hemingway forget the quote he JUST posted in the preceding paragraph?

As to the issue of the "myth of judicial supremacy," The AP may be treating the Supreme Court as a consensus opinion with the final say over whether or not Obamacare is Constitutional. In effect, they may be merely reminding people that Obamacare has not been ruled upon by the Supreme Court so it should not be considered a legal fact that Obamacare is unconstitutional, just personal opinion. I will admit I personally do not like the way the AP worded the phrase. It is definitely over-simplified, but Hemingway has failed to justify calling it "amateur-hour excusion into legal theory."

In addition, it should also be noted that when a person makes a statement that can be reasonably verified as true or false, it ceases to just be an opinion, but an actual FACT. And just because a claim appears on the Opinion page does not mean it is just opinion. It is common for writers to present facts on the Opinion page to help justify their opinions. If I said the earth was flat on an Opinion page, does that make my claim any less false? As I have noted before, Fact Checkers have to assess these claims based on how they would expect a reasonable person would interpret the claims. As a result, the AP provided information enough to cover many reasonable interpretations.

“The AP also did an extensive investigation into Obama’s handling of the Gulf spill, and concluded it ‘shows little resemblance to Katrina,’ ” writes Sargent. “As [liberal Washington Monthly blogger] Steve Benen noted in lauding this effort, the AP definitively debunked a key media narrative as ‘baseless.’ ” One could ask whether the BP oil spill was being compared with Katrina simply because of its relative proximity and public opinion that the Obama administration handled the crisis similarly poorly. But why bother? The very idea of fact checking a broad comparison should send readers who give a damn about facts screaming for the exits.

Unlike checking individual facts, broad comparisons create a much larger task for a fact checker. Although one could theoretically find a near unlimited set of potential comparisons, the overwhelming majority of these will likely be trivial. As a result, many broad comparisons can be reduced to a small manageable set of significant comparable facts. When this is the case, broad comparisons are actually quite useful. For instance, it is possible to compare significant aspects of Romneycare and Obamacare to determine just how similar those bills actually are. It is also possible to determine if comparisons of a given politician to Hitler are justified. To be fair, some broad comparisons cannot be reduced to a manageable set of significant check-able facts. However a reader who runs "screaming for the exits" just because they are confronted with a broad comparison has likely fallen into the fallacy that all comparisons not comprised of a single fact must be incomparably large. Such a fallacy has no place in the practice of Fact Checkers.



To be continued in part 3 of "The Weekly Standard Attacks Fact Checkers"

The Weekly Standard Attacks Fact Checkers (Part 1)

This is part 1 of a 3-part series entitled "The Weekly Standard Attacks Fact Checkers"

I am no stranger to The Weekly Standard's attacks on non-partisan fact checking sites. Recently, a relatively lengthy 3-page article caught my attention. The article, entitled "Lies, Damned Lies, and ‘Fact Checking’," is another in a long running series of articles attempting to poison the well, and keep people from engaging in critical thinking with non-partisan fact checking sites. Online editor Mark Hemingway is the author of this 'masterpiece.'

Most of this article is a collection of unsubstantiated claims and slippery slope fallacies. However, they did attempt to give a few examples to help substantiate a few claims. I thought it would be helpful to take a look at them. Unfortunately, these examples fall short of proving anything other than Hemingway's desperation to challenge the credibility of neutral fact-checking sites.

TO READ IS TO FLY

Hemingway starts with a look at an AP Fact Check of the November 12, 2012 Republican presidential debate:

On Iran, former Massachusetts governor Mitt Romney suggested that the U.S. government should make it “very clear that the United States of America is willing, in the final analysis, if necessary, to take military action to keep Iran from having a nuclear weapon.” Little did Romney realize that the AP is the final arbiter of America’s tactical military capabilities and can say with certainty that a military attack on Iran’s nuclear program should not be attempted. [AP:] “The U.S. certainly has military force readily at hand to destroy Iran’s known nuclear development sites in short order. This is highly unlikely, however, because of the strategic calculation that an attack would be counterproductive and ultimately ineffective, spawning retaliation against U.S. allies and forces in the region, and merely delaying eventual nuclear weapons development.”

Interestingly enough, the AP never used the words "should" nor "certainty." The word "should" only really exists in normative statements. However, it should be clear the statement was purely a positive (descriptive) statement, which should have been clear given the fact that the phrase used the keyword "is." It should also be noted the phrase "highly unlikely" should indicate that the AP is not "certain" either. Sadly, reading comprehension may be lost art.

Also fortunate for the savvy news consumer, the AP apparently has a better grasp of what America’s intelligence agencies do and do not know than Newt Gingrich, a man who used to be third in line for the presidency and has received countless classified intelligence briefings.

This is a stereotypical Argument From Authority, which is a fallacy in this case since Gingrich has been out of office for over a decade. Do I need to say that a lot has happened in both the fields of national defense and intelligence since then?
I will however point out that I am no fan of the AP's quick-response method. Is this what defense officials are saying? Experts? Additional details would be useful (although it may be less efficient).

At the debate, Gingrich suggested that there was room for improvement at America’s intelligence agencies, and noted in particular that we don’t have a reliable intelligence operation in Pakistan. The AP sprang to the defense of the CIA: “The U.S. killing of a succession of al Qaeda figures in Pakistan, none more prized by America than Osama bin Laden, demonstrates that the United States indeed gets vital and reliable intelligence out of Pakistan. While it may have been true when Gingrich left government in 1999 that the CIA’s spy network was limited, since 2001 the agency has dramatically expanded its on-the-ground operations worldwide,” the AP “fact check” concluded. The fact that bin Laden, the most wanted man on the planet, was living in a compound in Pakistan possibly for years may seem like a sign that our intelligence sources in the country leave something to be desired​—​but guess again, Newt.

Hemingway seems to be over-simplifying the quote a bit. Here is what Gingrich said: “We don’t have a reliable intelligence service. We don’t have independent intelligence in places like Pakistan. We rely on our supposed friends for intelligence. They may or may not be our friends. And the amount of information we might or might not have, might or might not be reliable.” As you can see, this is not just saying there is "room for improvement at America’s intelligence agencies." Gingrich is making the specific claim that that intelligence service in Pakistan is unreliable. What Hemingway fails to mention is that the AP partly AGREED with Gingrich: "Gingrich is right that Pakistan’s intelligence agency is an often-unreliable U.S. partner and elements of the country’s power structure have supported U.S. terrorist enemies. But as the bin Laden raid shows, the CIA is hardly impotent in its ability to operate alone in Pakistan." It is clear that the AP considers Gingrich's statement true, to a point. However they also wanted to let readers know that US Pakistani Intelligence is not completely unreliable. A Hemingway response of "probably" with absolutely no context towards intelligence reliability hardly challenges that claim. In fact, since the AP called US intelligence "often-unreliable," it would seem that the AP doesn't fully disagree that "our intelligence sources in the country leave something to be desired​." Does Hemingway realize not all Fact Checks produce the equivalent of a "false" rating? Here we have yet another case of poor reading comprehension.

To be continued in part 2 of "The Weekly Standard Attacks Fact Checkers"  

Note: As you may have noticed, I am no longer using the "Defending The Truth-O-Meter" series title. It made titles too long and I decided it was unnecessary.

Politifact Bias (or not) [Defending the Truth-O-Meter]

The Blog "Politifact Bias" seems to be a pretty popular blog for conservatives who want to confirm their poorly-justified belief that the neutral fact checking site known as Politifact is actually left-biased (at least according to Google Search Results). I have yet to look at this blog directly since they rarely publish original posts. However, I thought I may as well check out their "PolitiFact 2011: A review" post to get an idea of what they considered to be significant evidence of the diagnosed liberal bias of Politifact in 2011.¹ As far as I can tell, this blog merely collects criticisms of Politifact they find around the internet. In my experience, these criticisms can be extremely poor. They listed "a rundown of the issues that should keep discerning readers from trusting PolitiFact:"

1)  PolitiFact persistently ignores the effects of selection bias.  It simply isn't plausible that editors who are very probably predominantly liberal will choose stories of interest on a neutral basis without some systematic check on ideological bias.  PolitiFact, for example, continues to publish candidate report cards as though selection bias has no effect on the report card data.

Basically, the linked article⁴ is a great example for a subject I was planning on writing about, entitled "The Wrong Ways To Detect Bias." One of the methods I was planning on talking about was exactly what these people did. They just compiled a list of articles on Politifact over the course of a year and came to the conclusion that, since the worst ratings tend to favor republicans more than democrats, they must be biased in their selection in favor of the left. The problem with this is that their conclusion simply does not follow. Their are other possible reasons for this kind of result that have nothing specifically to do with Politifact being biased.

1. To give them the most credit, it is possible there may be some liberal selection bias at work here, but not in the way they think. Politifact often gets ideas for statements to check from their readers. It is possible most of their readers are liberal, which souds quite plausible given the fact that conservatives seem so ready to accuse Politifact of bias. This is no fault of Politifact, nor does it show any bias on their part. The fault lies squarely with Republicans who chose not to submit statements to Politifact and instead waste their time complaining about perceived bias (a vicious cycle). It should also be noted that the definition of Selection Bias is "A type of bias caused by choosing non-random data for statistical analysis." Well duh! The article itself mentions that Politifact Bill Adair noted "We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." That right there should show that there is selection bias, though not liberal selection bias. However, the article intends to show that the bias is toward the left, instead of toward important statements. If the article was intended to merely accuse Politifact of selection bias, they would basically be attacking a straw man since the main goal of Politifact is not to tally true and false statements to figure out which candidates are more honest (although this does happen occasionally). It should also be noted the selection of Politifact articles used in this calculation are very much non-random. As the article mentioned, the time period selected was when Democrats were in power. This could have easily affected the results, as I will get into later.


2. Could it be that Republicans were telling falsehoods more often during that time period? The article does mention this. "One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site. However, there is no evidence offered by PolitiFact that this is their calculus in decision-making." Yet they don't look for evidence that their explanation is the better one. To show the fallacy in this kind of thinking, let me present an analogy: Let's say i have two dogs that roam free in my house all day, Rex and Dex. I come home one day to find my couch clawed to pieces. I cannot find evidence that Rex did it, so I assume Dex did it (even though I found no evidence specifically pointing to Dex). This is exactly the same fallacy they employ. They don't specifically have evidence of a Republican tendency towards falsehoods so they assume it must be a Politifact tendency towards liberal bias. Like I noted earlier, the article did not randomly chose articles. Could it be that, since the Republican Party was out of power, they had a greater tendency towards falsehoods? Could it be that the ratings success of Fox News created a podium for more Republican falsehoods? Could it be that, with the rise of the right-wing Tea Party, extreme views have tended to produce more falsehoods? Could it be that the possible rise of anti-intellectualism in the Republican Party has made it more likely to back falsehoods? Could it be that the Republican Party itself is more prone to being dishonest? Could the Republican party be more ideological and less prone to checking facts that could challenge their ideology? Could it be that the Republican tendency to call neutral fact-checking sites "biased" creates a tendency to play fast and loose with the truth? These are all possibilities (although I am not necessarily claiming any of them to be true). So did the article control for or check any of these? NOPE. They could have checked other neutral fact-checking sites to see if they came to similar conclusions as Politifact. They could have compared these results with articles from before Obama was elected. They just dismissed these hypotheses for no better reason than they would have for dismissing their own hypothesis.


3. It could be that the significance of the truer statements was much higher than that of the false statements. Not all statements are equal. Chris Christie's statement about the city of Newark's graduation rate has far less of an impact on party image than the Democratic claims about Republicans trying to end Medicare. An overall look at a large number of graded statements can give you a decent look at a candidate's honesty, but you need to actually check the statements to see how significant they are. If Politifact is giving Democrats more high-profile false's then it doesn't matter that there are fewer. The primary focus of Politifact once again is to read the about ratings they give these statements. So a handful of high profile falsehoods for Democrats can look much worse than a bunch of inconsequential falsehoods for Republicans.

It should be pretty clear that merely tallying Politifact ratings doesn't give you anything remotely conclusive about the theoretical bias of a site.

2)  PolitiFact continues to publish obviously non-objective stories without employing the journalistic custom of using labels like "commentary," "opinion" or even "news analysis."  Readers are implicitly led to believe that stories like an editorial "Lie of the Year" selection are objective news stories.

Any evidence for this? A link or two maybe? If they wrote an article about this, it would be useful at this point. I may search for an article in their archives and come back to this later. As it stands, this is an unsubstantiated statement. I will, however, agree that the "Lie of The Year" title is a bad title since there is a difference between people who lie (know their statements are false) and people who are careless with the truth (don't necessarily know their statements are false). However, Politifact does have some leverage here since the 2009, 2010, and 2011 Lies of the Year have been tackled by Politifact, as well as other Fact Checking sites, so often, it is hard to imagine these politicians do not know their claims have been debunked. It is also good to note that Politifact lets readers know the "Lie of The Year" is just a consensus opinion of their editors and reporters, not objective news. So readers are actually not led to believe the Lie of The Year is an objective news story.

3)  PolitiFact continues to routinely apply its principles of analysis unevenly, as with its interpretation of job creation claims (are the claims assumed to refer to gross job creation or net job creation?).

The link provided goes to a re-post of this exact same blog entry. I'm interested to see what they are talking about. If I find evidence I will post my critique.³

4)  PolitiFact has yet to shake its penchant for snark.  Snark has no place in objective reporting (see #2 above).  Unfortunately, PolitiFact treats it like a selling point instead of a weakness, and PolitiFact's intentional use of it has apparently influenced Annenberg Fact Check to follow suit.

I'm not 100% sure what he means by "Snark"² However, I will assume he means "Use of sarcasm or malice in speech." I'm not sure I have ever seen sarcasm. But I fail to see the problem here. Politifact is in the business of spreading awareness of "Fact Checking." It helps keep readers interested and coming back for more (although it's nowhere near as common as in Glenn Kessler's articles). Overall, it has no negative effects on the actual research. In fact, it can help get readers fired up about falsehoods and dishonesty. Other than personal taste, I see no problem with this. In fact, any case of "snark" I have ever seen has been quite subtle, meaning I cannot see it honestly turning away any readers that care more about reality than ideology.

It seems as though none of these "issues" should "keep discerning readers from trusting Politifact" so long as they care about quality fact-checking to help ensure they don't subscribe to false beliefs. Only one of these "issues" even had a linked article, and the content of that linked article came nowhere near demonstrating bias at Politifact. I will empathize that showing the existence of bias is no easy task (usually). But if you are having THAT much trouble exposing bias, maybe you should wonder if the bias really exists at all...


¹ - Update 1-10-12: The writers of "Politifact Bias" have informed me that this post was meant to be just a summary, and was not meant to give significant evidence of bias. However, this does not excuse the fact that they did list "the issues that should keep discerning readers from trusting PolitiFact," without providing evidence for a few of them. This does not change the rest of this post because all this post does is check to see if these "issues" significantly exist at all.

² - Update 1-10-12: The writers of "Politifact Bias" have informed me that, by snark, they mean that Politifact is conveying their own biases. When I asked for an example, they provided me with a quote from a July 2011 politifact article talking about Newt Gingrich:"With the 2012 election in their sights, Republican candidates spend most of their time trying to prove that President Barack Obama and the Democrats will make the economy worse." However this is a terrible example given that this is not an uncontroversial fact. There is no hint of the reporter's feelings, biases, or prejudices. Nor is there any hint of unnecessary partisan adjectives. It sounds as if this example fails to substantiate their claim that Politifact is guilty of "snark." 

Update 1-10-12: I have edited this post to enhance readability. No significant changes were made to the content.
 
³ Update 1-10-12: A writer of "Politifact Bias" has informed me that the link was to a keyword search on one of his blogs (seems a bit unnecessarily distracting and cryptic). He verified a certain post did a good job underscoring his complaint. In this post he mostly just lists each case and states whether it was helped or harmed by the inclusion/exclusion of "gross jobs." The problem is he mostly fails to investigate context. There are some statements where it is appropriate to look at net jobs, such as a statement that looks at "jobs added" as a whole. Conversely, there are statements where it is appropriate to ignore lost jobs. When a politician states the stimulus did not create one job, it doesn't matter how many jobs were already lost as a result of the recession, just so long as at least one job was directly created by the stimulus. Since the mechanism for lost jobs and the mechanisms for gained jobs are different, it is more than appropriate to ignore what has nothing to do with the mechanism under question. For example, If I went to a village under attack by militants and pulled a single child to safety, thereby saving it's life, should I not be credited with saving that kid's life just because more than one life was lost in that village from the attack? Any reasonable person would find this absurd. Granted he attempted an investigation into the context of choosing based on the word "created." I will give him some credit for that, but he did not investigate whether or not Politifact was consistent when the mechanism that "created" jobs was the same as the mechanism that caused jobs to be lost. He just simply focused on the word "created." As the stimulus example shows, his list definitely ignores important context. It should also be noted that Politifact has actually made the context used in their decision making well known (see last link).

⁴ - Update 1-16-12: I have also completed a comprehensive critique of this article as well.