The Downside of Rating Systems In Political Fact Checking

In his farewell message to fans of political fact checking, departing FactCheck.org director Brooks Jackson reflects on the growth of the fact checking industry, the merits of fact checking, criticisms of fact checkers, and various legitimate pitfalls made by various fact-checking sites. Among the pitfalls he discussed, the one that most caught my eye dealt with the ratings systems fact checkers use so often:

Rating statements with devices such as “truth-o-meters” or “Pinocchios” are popular with readers, and successful attention-grabbers. But such ratings are by their nature subjective — the difference between one or two “Pinocchios” is a matter of personal judgment, and debatable. Some statements are clearly true, and some provably false, but there’s no agreed method for determining the precise degree of mendacity in any statement that falls somewhere in between. Rating systems have also led to embarrassment. A senator who said a “majority” of Americans are conservative was rated “mostly true” (and later “half true”) even though the statement was false. The story cited a poll showing only 40 percent of Americans rated themselves conservative. That’s more than said they were moderate (35 percent) or liberal (21 percent) but still far from a majority. The senator had a point, but stated it incorrectly, thereby exaggerating. A simple “truth-o-meter” had no suitable category for that. Our approach would have been to say that it was false. But we would also note that the senator would have been correct to say Americans are more likely to call themselves conservative than moderate, or liberal, when given those three choices.

While I disagree that ratings systems are entirely subjective (most have specific rules for each category), I do agree that the organization of categories is not rigorous. There are obviously statements that do not easily fit into any given category. Furthermore, these rating systems can be a distraction, giving the reader an incentive to merely look at the rating and ignore the actual fact checking. Although I do understand not everyone has time to read an tire article over every claim that has been checked, simple summaries (such as the ones used by FactCheck.org) at least give the reader a basic idea as to what was right and/or wrong with the checked claim. I will admit ratings systems have doubtless contributed to the rising popularity of fact checking. But it isn't clear whether or not they actually do more harm than good.

In addition to discussing the pitfalls of fact checking, Jackson also made some very good points about the actual purpose of fact checking in political discourse:

"Complaining that fact-checkers failed to stop politicians from lying is like complaining that a firefighter failed to prevent an arsonist from starting a fire.

Furthermore, it seems to me that anyone who asks the very political operatives behind the 2012 falsehoods to rate our performance is pretty much interviewing the arsonists about the merits of the firefighters. We don’t write to impress politicians or their hirelings. We write to help the voters — and we don’t expect to get an invitation to dinners at the White House. We can’t stop politicians from trying to bamboozle voters. But we can make voters harder to fool."

Indeed there is an extremely important role for fact checkers to play in political discourse. And Jackson sums it up quite nicely. Scientific skepticism, which fact checkers apply to politics, has a role in nearly every aspect of life, including politics. In this spirit I thank Mr. Jackson for the quality work both he and his team have done over the past nine years. FactCheck.org is my favorite fact checking site and I wish him the best of luck in the future!

Why Are The Washington Post Wonkbloggers Repeating A Romney Campaign Falsehood About Medicare?

The Washington Post’s WONKBLOG has been repeating the Romney Campaign's falsehood about Obamacare and Medicare. From their fact check of Bill Clinton's 2012 DNC Speech:

FALSE: [Clinton:]“Both Governor Romney and Congressman Ryan attacked the president for allegedly robbing Medicare of $716 billion. That’s the same attack they leveled against the Congress in 2010, and they got a lot of votes on it. But it’s not true.”

The Affordable Care Act did indeed cut Medicare spending by $716 billion, as the Congressional Budget Office wrote in a July 24 report. It does that by reducing payments to Medicare hospitals and doctors, essentially ratcheting down the amount they receive when they see a patient.

...these Medicare cuts do indeed exist.

However, all three major fact checkers have come to pretty much the opposite conclusion:

• FactCheck.org (here, here, and here) 
• Washington Post Fact Checker (here and here)
• Politifact.com (here, here, here, and here):

In my post on Ryan’s Medicare claim, I explained why this is the case using the findings of both FactCheck and The Washington Post Fact Checker (PolitiFact did an exceptionally poor job explaining the problem):

"Medicare payroll taxes go to pay for Medicare Part A. What revenue is left over goes into the Medicare Hospital Insurance trust fund. And that fund goes to finance shortfalls in Medicare Part A. As of late, there is not enough revenue coming from payroll taxes to cover hospital insurance for medicare beneficiaries. As a result, the trust fund is being drained and will eventually be empty, meaning Medicare Part A will become insolvent. To help mitigate the problem, the Affordable Coverage Act slows the amount of money coming out of the trust fund via reduced payments to hospitals. Obama can get away with those reduced payments to hospitals largely without causing them to drop out of the program because of a deal he stuck with hospitals, promising new patients through the ACA, more than making up for the lost revenue. This saved money stays in the trust fund (although it can be loaned out, more on that later). It does not go to cover the uninsured not currently in Medicare. In addition, Obama increases the revenue coming into the trust fund via a 0.9 percent tax "on earnings above $200,000 for single taxpayers or $250,000 for married couples."

The savings from Medicare that actually go into Obamacare come from Medicare Parts B, C, and D, and only from funds that come from congress (income taxes, corporate taxes, etc...) to cover outlays. This accounts for approximately 2/3 of $716 Billion in slowed medicare growth. Obama decreases the growth of money congress spends on Medicare from general revenue using the Independent Payment Advisory Board, which is incredibly limited in what it can target."

...a bill can both reduce the deficit and not be "paid for." This is because some of the revenue that comes in goes "into savings" instead of being used to pay the bills. And just like a bank uses savings accounts to make loans, so too can this Medicare money "in savings" be loaned out to pay for other programs. FactCheck explains:

"What happens in government accounting is that after Treasury issues a bond that it will have to pay later, it can spend the money it received on other things. And it often does, whether that’s coverage expansion, as called for in the health care law, or any number of things. The Romney spokesman said it was “a shell game.”"

But that doesn’t mean that the Medicare trust fund will be slashed or its “piggy bank” “robbed” or “raided,” or any other claim we’d put firmly in the category of “senior scare.” (emphasis mine)

Romney does have a point in essentially calling this an accounting trick. And he could spend time making that claim instead of a demonstrably false one. You can argue over whether or not the CBO was therefore wrong to say the ACA cuts the deficit. But you cannot say payroll taxes are going toward the ACA anymore than you can say your personal savings in the bank are going toward someone else's loan. Just like those savings are still legally yours, so are the trust funds still legally Medicare's. Any money that is borrowed from the fund has to be paid back.

I am not quite sure why the Wonkbloggers are doing this. And I can find no explanation as to why they keep coming to the exact opposite conclusion as these major independent fact checkers, including the Washington Post’s own Glenn Kessler. Could It be the double counting? Romney could certainly say Obamacare isn’t paid for, or that it adds to the debt (but not deficit). However, as Glenn Kessler has noted, that would open Republicans (including his own VP Pick) to criticisms of hypocrisy since those claims can be leveled at nearly any deficit reduction plan, including many of the GOP's own.

The Roundup: Abortion Edition

Todd Akin is taking the GOP down with him.

Yes, Plenty of Republicans Want to Ban All Abortions, Full Stop
The Todd Akin incident reminds us that this election could easily decide the future of a woman's right to choose. Multiple states have attempted to pass bills that could effectively ban all abortions if Roe v Wade was overturned (thanks to a conservative appointee from a Romney administration and the inevitable GOP controlled senate).

PolitiFact: Women's group says Paul Ryan would "outlaw in vitro fertilization”

"UltraViolet has a point that the bill Ryan backed could significantly alter the way in vitro fertilization is practiced. However, the group exaggerates when it says the bill in question would "outlaw in vitro fertilization." The bill doesn ’t outlaw the procedure directly -- whatever impact it has would likely require action by states, which many states may be unwilling to undertake. And while the bill likely outlaws specific practices that have historically been considered important for practicing in vitro fertilization, it would not ban the procedure itself."

FactCheck: Another Abortion Falsehood from Obama’s ‘Truth Team’
Some clarification on the 2012 GOP platform on abortion:

"It’s true that Romney has voiced support for the 2008 platform’s call for an unspecified “Human Life Amendment” to the Constitution, and the language approved by the party’s platfor m committee for 2012 is identical. But that’s a far cry from advocating an abortion ban that would apply in cases of rape or incest."

And it turns out that we have been to careless over speculation that these Human Life Amendments will ban abortions in the cases of rape and incest:

"As we’ve said before, there have been numerous versions of human life amendments proposed over the years, some of which include exceptions for rape and incest and some of which don’t. For details, see our July 31 item, “Falsifying Romney’s Abortion Stance, Again.” Most of these amendments didn’t get out of committee."

However, the speculation is not entirely unfounded:

"Furthermore, Romney’s chosen running mate, Paul Ryan, opposes exceptions for rape or incest. And it would be accurate to say that the GOP platform calls for a constitutional amendment that would leave states free to adopt abortion bans without exceptions."

So Red States would be screwed still....

However, as Suzy Khimm of The Washington Post finds, it may also be the case that none of this matters.

"But do party platforms even matter? Much of the political science research suggests not — at least when it comes to the candidates’ own views and actions. “The nominee is not necessarily constrained by the formal platform. They can agree with whatever bits and pieces and ignore the rest,” says John Sides, a political science professor at George Washington University."

Imagine You Were Raped. Got Pregnant. Then Your Rapist Sought Custody.

"The debate over Rep. Todd Akin's widely condemned comments on "legitimate rape" has largely centered on abortion and Republican efforts to outlaw the procedure, even in cases of rape. But the controversy has also uncovered a little-discuss ed issue: When some rape victims do choose to give birth to a child conceived through sexual assault, they find that the legal door is left wide open for their victimization to continue. It sounds unfathomable, but in many states the law makes it possible for rapists to assert their parental rights and use custody proceedings as a weapon against their victims."

The Doctor behind Akin's offensive unscientific statements

Update 8/27/12:

The Weekly Standard Defends Ryan on Redefining Rape

"More broadly, the "they only intended to exclude statutory rape" defense misses the point. Most serious abortion foes oppose the rape exception—full stop. It's no surprise that the abortion rights opponents who wrote H.R. 3 didn't foresee that simply narrowing the rape exception—a move far short of their preferred position—might provoke more controversy than opposing the rape exception itself."

...

"While I was reporting out the forcible rape story last year, one of the sources I called, a very accomplished woman, told me she had been a victim of statutory rape as a young teenager. Decades later, she nearly broke down about it on the phone while talking to a stranger. Should she have been denied the option of a Medicaid-funded abortion because her rape wasn't rapey enough?" (emphasis mine)

No PolitiFact Texas, The Texas GOP IS Calling For Schools to Stop Teaching Critical Thinking Skills

As readers of this blog know, I am a frequent defender of independent fact checking operations such as PolitiFact, FactCheck, and The Fact Checker. These operations are necessary to promote critical thinking and reality based political discourse. However, the people who run these operations are human and do occasionally make mistakes (although quite a bit fewer than some would have you believe). I believe a recent PolitiFact Texas article is indeed guilty of one of these mistakes. The article examined a claim by New York Times columnist Gail Collins in which she claims "The current Republican state [Texas] platform calls for an end to the teaching of ‘critical thinking’ in public schools." PolitiFact Texas argues this was taken out of context and assigns a rating of "Half True" to the claim. But did PolitiFact do a satisfactory job justifying this rating?

As PolitiFact notes, Outcome Based Education is not a concrete idea.

"So what the heck is it? The news stories we read indicate outcome-based education takes different forms nearly everywhere it's applied. The Fort Worth Star-Telegram gave a description in an Oct. 30, 1996, news story about opposition to OBE-like elements in the state's planned education overhaul:

Under outcome-based education, academic and personal goals are set for students before they can graduate. The program stresses that children are not allowed to fail, so they might be given the same test or report over and over until they do the work satisfactorily. It also may eliminate traditional grades, competitive student assessments and distinct subjects and grade levels.

Methods of implementing outcome-based education include awarding group grades instead of individual grades and eliminating honors programs.

The "founding father" of OBE, education reformer William Spady, gave an example in an interview for the December 1992/January 1993 issue of Educational Leadership magazine, published by ASCD (formerly the Association for Supervision and Curriculum Development). Asked whether an outcome might be "The student will be able to list the five causes of the Civil War," Spady replied: "No, sorry; that is not an exit outcome. But, 'Identify and explain the fundamental causes and consequences of the Civil War' would be an enabling outcome worth pursuing en route to some larger exit outcome.""

That is interesting, but could that be what the Texas GOP is actually referring to? PolitiFact takes a look at their 2010 platform:

""The primary purpose of public schools is to teach critical thinking skills, reading, writing, arithmetic, phonics, history, science, and character . We oppose Outcome-Based Education (OBE) and similar programs.""

Based on what PolitiFact has found as a description of OBE, nothing in the Texas 2010 platform seems to apply here. In fact, the earlier Civil War example sounds like an attempt to actually understand the civil war instead of just memorizing a few facts. It is the difference between telling a student to "list" and "explain." The former is mere memorization. The latter includes thinking critically about the causes and outcomes. So why again does the GOP not support OBE? Let us look again at their 2012 platform:

"We oppose the teaching of Higher Order Thinking Skills (HOTS) (values clarification), critical thinking skills and similar programs that are simply a relabeling of Outcome-Based Education (OBE) (mastery learning) which focus on behavior modification and have the purpose of challenging the student’s fixed beliefs and undermining parental authority." (emphasis mine)

Humans are not natural critical thinkers, so behavior modification is necessary for critical thinking. And critical thinking necessarily also involves challenging fixed beliefs and authority. So it sounds as if the reason the Texas GOP wants to ban OBE is actually because of qualities necessary for critical thinking. If they want to ban qualities necessary for critical thinking, does it not follow that they must therefore want to effectively ban critical thinking all together? Donald Prothero of SkepticBlog explains:

And the last phrase is almost too bizarre for belief: they explicitly don’t want their kids to think critically because their ideas might challenge “the student’s fixed beliefs and undermine parental authority.” That’s it in a nutshell—the classic dogma of fundamentalist churches and authoritarian states. Don’t think for yourself, don’t ask questions, don’t rock the boat, because the Church, the GOP and your parents are the ultimate repository of truth, and we don’t want to have the lies we told you as children undermined by anything like education.

So PolitiFact is correct that critics have failed to take note of the OBE part of the GOP platform. But given how they describe OBE, it is clear they are talking about what essentially boils down to critical thinking. So the distinction should have made this article score a "Mostly True" at worst.

In addition it is striking is that PolitiFact falls for the GOP's weasel word use of the term "critical thinking":

"Both [2010 and 2012] platforms support critical thinking when it comes to "controversial theories" such as evolution, which "should be taught as challengeable scientific theories ... Teachers and students should be able to discuss the strengths and weaknesses of these theories openly and without fear of retribution or discrimination of any kind.""

There is strong evidence to suggest "critical thinking" in this context has little to do with actual critical thinking skills. It is essentially used as a weasel word for unsubstantiated challenges to the theory of evolution and climate change. The fact that the Texas GOP called the theory of evolution "controversial" speaks volumes as to what they are saying. The theory of evolution is accepted by nearly all biologists and scientists all over the world. It is far from controversial outside the minds of a generally scientifically illiterate public. Any kind of actual controversy within the theory of evolution is far above the level of any high school student. The "controversies" they speak of are actually a large number of widely debunked creationist arguments unfit for science education. This is not critical thinking. Just because the Texas GOP calls it critical thinking doesn't mean they support critical thinking skills. This is an obvious fallacy. For example, I could say I support a good diet but claim people should eat at McDonalds every day. Do I really support a good diet? Would someone not be justified in claiming I do not actually support a good diet.

So it sounds as if PolitiFact has missed the point. But this is to be expected every once in a while. Although PolitiFact is a great source of information in a world filled with disinformation, they are not always right. Should you dismiss them because they are not always right? Of course not! You would have to dismiss all information because the people that produce that information are all imperfect. PolitiFact has an excellent track record. So chances are their analysis is correct. All this post does is reinforce what fact checkers such as PolitiFact (as well as skeptics in general) have tried to say. Think critically and don't take anything for granted. Do not pick and chose what you are skeptical of based on pre-existing biases, and know when to reasonably accept a claim. Do not be a denier.

The Roundup: Romney's Bain Edition

Romney tells Obama to stop sharp political attacks
I find it ironic that Romney accuses Obama of lying, while following up with lies of his own.

And Romney continues to lie...
 
WP Fact Checker: 4 Pinocchios for an unproven Romney claim of ‘crony capitalism’

FactCheck:‘You Didn’t Build That,’ Uncut and Unedited

WP Fact Checker: Romney’s claims of misspent stimulus money 
 
WP Fact Checker: An Obama quote taken out of context, yet again

Politifact: Mitt Romney says a lawsuit filed by President Obama's campaign challenges voting privileges for the military

FactCheck: Does Obama’s Plan ‘Gut Welfare Reform’?

Note that half of these fact checks come from the same fact checker Romney cited when demanding an apology from the Obama campaign...

An amazing take-down of the ideas that underlie Romney's economic and fiscal agenda.


Mitt Romney and Bain: a Fact Checker collection  
 After citing Kessler in an a calling Obama dishonest for the bain claims, Romney and Bain fail to provide evidence showing he had "no role whatsoever" in Bain between 1999 and 2002. As a result, the Fact Checker is at an impasse. Romney's secrecy continuing to become a major problem for him.
 
David Frum: With nothing else to do, Obama calls Romney a criminal or a liar

"Romney’s best biographers, Michael Kranish and Scott Helman (both of them also Globe reporters), support Romney’s version of events. They describe Romney’s departure from Bain as so sudden and total as to have nearly capsized the company.

On the other hand, he did keep signing forms and cashing checks. Also on the other hand, the harsh things that happened post-1999 followed from decisions made pre-1999."

David Frum: Romney: Too Weak?
Is David Frum going to re-think his vote this November?

"at every point, Romney has surrendered to the fringe of his party. Weak. And now in his first tough encounter with Barack Obama, Romney is being shoved around again. This is not what a president looks like - anyway, not a successful president."

Paul Krugman: The Class-Warfare Election

"So like it or not, we have an election in which one candidate is proposing a redistribution from the top — which is currently paying lower taxes than it has in 80 years — downward, mainly to lower-income workers, while the other is proposing a large redistribution from the poor and the middle class to the top."

Paul Krugman: Pathos of the Plutocrat

"Clearly, Mr. Romney believed that he could run for president while remaining safe inside the plutocratic bubble and is both shocked and angry at the discovery that the rules that apply to others also apply to people like him. Fitzgerald again, about the very rich: “They think, deep down, that they are better than we are.”"

The Creepy Small Lies of the Romney Campaign

"If Republicans are willing to just flat-out lie about what he said, it's impossible to self-edit your remarks enough to avoid it. We've now seen the Romney campaign make hay out of three wild misquotations"

"I know I keep asking this, but has any previous campaign ever done this on such a routine basis? I don't mean to suggest that no campaign has ever been as nasty. Obviously Willie Horton and "creating the internet" and the Swiftboating of 2008 were worse. And both sides traffic in distortions and cherry picking all the time. But there's something about the methodical small lies of the Romney campaign that seems quite new." (emphasis mine)

Romney’s World: Mitt’s insults, mistakes, and blunders abroad aren’t gaffes. They actually represent his true worldview

"The thing that Krauthammer doesn’t get is that Romney is not the sort of businessman—that his brand of capitalism is not the sort of enterprise—that requires even the most elementary understanding of diplomacy, courtesy, or sensitivity to other people’s values, lives, or perceptions."

They Also Serve Who Play Whack-a-Mole with People Who Do Not Understand the Gordon Equation for Valuing the Stock Market

"Suppose that you had told me, 15 or 30 years ago, that there was an economist who did not understand the Gordon equation for stock market valuation: somebody who, instead of knowing that the Gordon equation was P=D/(r-g) (where P is the va lue of the stock index, D is the dividend paid on the index, and r and g are the required rates of return and expected dividend growth rates respectively) thought instead that it was P=E/(r-g) (where E is the account earnings of the index). Suppose you had told me that that somebody would be a respected senior economic adviser to Republican presidential candidates. Suppose you had told me that that somebody would be taken seriously by the press as an authority. And suppose you had told me that I would feel compelled to play whack-a-mole, in a largely vain attempt to limit the spread of misinformation…

I would simply not have believed you…"

Walter Jones On Mitt Romney's Tax Returns: 'I Don't Think This Will Go Away'

"Many Republicans have called on Romney to release more than two years of tax returns to make the issue go away, including his former rivals Newt Gingrich, Rick Santorum, Rick Perry and Jon Huntsman. Conservative pundits Bill Kristol and Ge orge Will, as well as the editors of the National Review, have also called on Romney to release the returns."

PolitiFact: Mitt Romney says Barack Obama’s plan for welfare reform: "They just send you your check."

"Romney’s ad says, "Under Obama’s plan (for welfare), you wouldn’t have to work and wouldn’t have to train for a job. They just send you your welfare check."

That's a drastic distortion of the planned changes to Temporary Assistance to Needy Families. By granting waivers to states, the Obama administration is seeking to make welfare-to-work efforts more successful, not end them. What’s more, the waivers would apply to individually evaluated pilot programs -- HHS is not proposing a blanket, national change to welfare law."

And Obama's counter claim that Romney requested a waiver like this himself back in 05 is actually comparing apples to oranges.


WP Fact Checker: Romney’s economic scorecard: Promising success, comparing records

"The scorecard’s simple arrows don’t account for the fact that Massachusetts lagged behind the nation as a whole in certain areas of the economy — jobs, home values and family incomes — while Romney was in office.

The scorecard also hides the fact that some economic conditions deteriorated for Massachusetts during the candidate’s last year in office, as is the case with home values and family income.

As for the budget-deficit comparison, it suggests Romney had a choice about whether to balance the Bay State’s budget during each year of his administration. But Massachusetts law required him to do so.

Finally, the scorecard gives the president no credit for taking office in the midst of the most severe recession in modern times. Obama’s numbers are generally positive since the end of the downturn, even if the recovery has been relatively slow. Democrats would argue that Congress is as much to blame as the president for that.

Nothing on Romney’s scorecard is flat-out false, but just about all the comparisons either lack context or ignore facts that contradict its assertions. Overall, the Romney campaign earns two Pinocchios."

Mitt Romney: Not a Murderer

"The problem with the ad, rather, is that it relies on an argument Obama obviously doesn’t believe: that a business owner engaging in layoffs is morally responsible for what happens to his employees afterward."

PolitiFact: Harry Reid says anonymous source told him Mitt Romney didn't pay taxes for 10 years 
Reid needs to give up. You don't spread speculation like this, especially when the likelihood of it being true is so unbelievably low:

"Reid has produced no evidence to back up his claim other than attribution to a shadowy anonymous source. Romney has denied the claim, and tax experts back him up, saying that the nature of Romney's investments in Bain make it highly unlikely he would have been able to avoid paying taxes altogether -- especially for 10 years."

And from the WP Fact Checker:

"In other words, this tax return shows a portfolio that is not structured to yield zero taxes. We spoke to a number of tax experts, all of whom said that, given Romney’s current portfolio, it was highly improbable for Romney to have had 10 years with taxfree returns — though there could have been one or two years with little or no taxes." (emphasis mine)

Is it just me or are the liberal claims about Romney's time at Bain beginning to look like the birther fiasco? 

PolitiFact: Checking the facts about Romney and Bain Capital  

"Bain and outsourcing are likely to remain sources of conflict between the candidates as the campaign goes on. It's ironic, then, that when it comes to the basic economics of outsourcing, the candidates basically agree—as do most economists . Many voters, however, see things differently—which is probably why neither of the candidates is talking about what they have in common."

The AP mentions fact checks that have debunked things Obama has said about Romney while neglecting to mention fact checks that debunk what Romney has said about Obama. What Liberal Media?

Stop celebrating Harry Reid for the same crap we berated the TEA Partiers for.

Obama Knocks Romney for Saying True Things About Coal

A Primer on Political Fact Checking: Index

more to come...

Politifact: Selection Bias or Bad Statistics

In a recent post where I critiqued a list of issues that should "keep discerning readers from trusting Politifact", I was alerted to an article examining potential selection bias in how Politifact chooses which statements to rate. The article, entitled "Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats," was written by Eric Ostermeier, "Research Associate at the Humphrey School's Center for the Study of Politics and Governance." This article has also been cited in a recent attack on fact checkers by The Weekly Standard (3-post critique: 1,2,3). Given the attention this article received, I thought it would be useful to see if this article actually gives any good reason to justify suspicion of selection bias at Politifact.
First off, it should be noted that selection bias occurs whenever a sample, taken for statistical analysis, has not been chosen randomly. It should be clear, given the title, as well as a few other indicators shown later, this article is focusing on party-related selection bias. However, we should not jump to too many conclusions yet. It is important to actually read the article to see if the title is even justified.

MISLEADING TITLES

" [TITLE:] Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.

[BODY:] PolitiFact assigns "Pants on Fire" or "False" ratings to 39 percent of Republican statements compared to just 12 percent of Democrats since January 2010 "

The difference between the title and body is subtle, but incredibly important. The title suggests the trend of "false" ratings is an overall trend throughout the 4 1/2 year history of Politifact. However, the actual article just looks at an 11 month period, mostly 2010. The reader should take note that this is not a random sample. In fact, as I will show later on in this article, this distinction actually seriously challenges any possible conclusion of party-related selection bias.

A BIT OF BACKGROUND

" PolitiFact, the high profile political fact-checking operation at the St. Petersburg Times, has been criticized by those on the right from time to time for alleged bias in its grading of statements made by political figures and organizations. "

At least in my experience, these allegations are often incredibly poor (also see linked critiques for analysis of other articles alleging bias within Politifact).

" The organization (and now its more than a half dozen state offshoots) grades statements made by politicians, pundits, reporters, interest groups, and even the occasional comedian (anyone 'driving the political discourse') on a six point "Truth-O-Meter" scale: True, Mostly True, Half True, Barely True, False, and Pants On Fire for "ridiculously" false claims.

But although PolitiFact provides a blueprint as to how statements are rated, it does not detail how statements are selected.

For while there is no doubt members of both political parties make numerous factual as well as inaccurate statements - and everything in between - there remains a fundamental question of which statements (by which politicians) are targeted for analysis in the first place. "

Politifact actually does now provide a description of how it chooses its statements:  

"Every day, PolitiFact staffers look for statements that can be checked. We comb through speeches, news stories, press releases, campaign brochures, TV ads, Facebook postings and transcripts of TV and radio interviews. Because we can't possibly check all claims, we select the most newsworthy and significant ones. 
In deciding which statements to check, we ask ourselves these questions:

• Is the statement rooted in a fact that is verifiable? We don’t check opinions, and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole.
• Is the statement leaving a particular impression that may be misleading?
• Is the statement significant? We avoid minor "gotchas"’ on claims that obviously represent a slip of the tongue.
• Is the statement likely to be passed on and repeated by others?
• Would a typical person hear or read the statement and wonder: Is that true?"

This page was not available when Ostermeier wrote this article, so I cannot criticize him for failing to notice it. However, it should be noted that Ostermeier's statement is no longer true (by no fault of his own, of course).

At this point, the article outlines its findings, drawing no conclusions at the moment.

IS IT APPROPRIATE TO DRAW CONCLUSIONS FROM 2010 ALONE?

Before we go any further, let us return to a previous point I wrote earlier. Remember the sample of posts from this article included all posts from January 2010 through January 2011. So we should ask ourselves if the chosen dates are truly arbitrary. To help answer this question,  I took a look, at data from 2007 to see if it differed from 2010 (note that when I say 2010 in this section, I'm including January 2011 to reflect Ostermeier's article):

 "Let's compare these results to the findings of Ostermeier for 2010:

•  Out of 39 statements rated "False" or "Pants on Fire," each main party received approximately 50% of the rulings. This is much different from 2010, where Republicans received 75.5% of these ratings.
• Politifact has devoted approximately equal time between Republicans (124, 52%)  and Democrats (110, 46%). This is nearly the same as 2010.
• Republicans were graded in the "False" or "Pants on Fire" categories 16% of the time (39% in 2010) while Democrats were rated in these categories 16% of the time as well (12% in 2010). This absolutely nowhere close to the "super-majority" found in 2010.
• 73% of Republican statements in 2007 received one of the top 3 ratings (47% in 2010) as opposed to 75% of Democratic statements (75% in 2010). When looking at "True" statements alone, Republicans beat Democrats with 5% more of their statements receiving the coveted award than Democrats. It is hard to see how one party was considered more or less truthful than the others. It depends on how much credit you give the "Half True" rating, as opposed to the "True" rating.
• Republicans received a slightly larger percentage of "Mostly False" ratings than Democrats (1.79%). This is the same as in 2010. However, this only results in a 2% difference between Republicans and Democrats for the bottom 3 ratings. This is MUCH different from the 28% difference in 2010. 

As you can see, the results from 2007 can seriously undermine many possible conclusions that a person could draw from the 2010 data."

OSTERMEIER STARTS POKING AT CONCLUSIONS

However, Ostermeier starts unjustly suggesting certain conclusions and not others.

" What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans.

And yet, PolitiFact chose to highlight untrue statements made by those in the party out of power. "(emphasis mine)

It sounds like Ostermeier has already come to the conclusion that Politifact is essentially guilty of party-related selection bias since they "chose to highlight untrue statements made by those in the party out of power." This does not necessarily mean Politifact is choosing these statements because they came from Republicans, just that they are the party out of power. The data from 2007 does not provide us with any hint as to the truth of this possibility. Neither party was really "in power" during that time. Democrats controlled both the House and Senate while Republicans controlled the White House. However, Ostermeier fails to ask a few questions:

• Do parties out of power tend to be  more likely to spread falsehoods, possibly in order to discredit the party in power (The party in power may not need to spread falsehoods as often since they are already in power)?
• Was there a sensational movement (such as the TEA Party) that may have caused  more air time to go towards the party out of power, increasing the chance of falsehoods (see first bullet)?
• Did the party out of power decide its number one priority should be to oust the current party from power? Do these kinds of campaigns tend to produce falsehoods?
• Did the party out of power feel the need to attack legislation (Obamacare) coming from the party in power, regardless of factual accuracy (See first bullet)?
• Was the party out of power often given uncritical air time from incredibly popular news outlets?

These are just some of the questions Ostermeier should have posed to readers before coming to the conclusion that Politifact "chose to highlight untrue statements made by those in the party out of power." Essentially, he neglected to ask if the party out of power (or the media) highlighted its own untrue statements?

OSTERMEIER DOES ACTUALLY MAKE A GOOD POINT

" An examination of the more than 80 statements PolitiFact graded over the past 13 months by ideological groups and individuals who have not held elective office, conservatives only received slightly harsher ratings than liberals. "

One would think this alone would challenge the suspicion of a liberal bias, maybe even the suspicion of Democrat-centered bias. Is Politifact biased against Republicans, but not Conservatives? That sounds like it would take quite a conspiracy theory-styled rationale to justify. In fact, could there be other possibilities that could explain this kind of trend in 2010?

•  Were there sensational primary elections within the Republican Party that may have contributed to extra coverage? Did this election deal with a bloc of furious voters, who may have been more prone to falsehoods?
• Could it be that at least a few former politicians became political commentators (also remember party out of power points)?

FINDING ANSWERS:

" When PolitiFact Editor Bill Adair was on C-SPAN's Washington Journal in August of 2009, he explained how statements are picked: 

"We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it."

If that is the methodology, then why is it that PolitiFact takes Republicans to the woodshed much more frequently than Democrats?

One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site. "

First off, it should be noted that Bill Adair has essentially admitted Politifact's selection of statements is not actually random. However, since his selection criteria does no appear to be dependent on political affiliation, the criteria can still be thought of as random in regards to it.

Ostermeier does suggest a possible reason for why Republican politicians are getting the brunt of bad ratings. In fact, I can imagine a Democrat saying Ostermeier's data  "shows that Republicans are more prone to falsehoods than Democrats." Now, as our 2007 data shows, whatever trend Ostermeier found in 2010 did not exist in 2007. We could ask ourselves if something changed in the Republican Party (like the TEA Party) that may have made the party more likely to embrace falsehoods. Notice that the stats for Democrat statements were the same in 2007 as they were in 2010. The big difference came in Republican statements, possibly suggesting a change in the Republican party. Now I'm not saying this is definitely the answer. However, I see no evidence that eliminates this possibility in favor of any possibility involving party-related selection bias.

" However, there is no evidence offered by PolitiFact that this is their calculus in decision-making. "

I'm not 100% sure why he says "However." If one political party is making "a disproportionately higher number of false claims than the other," wouldn't one expect a random large sample of those statements to also contain a disproportionately higher number of false claims (leading to "false" ratings from fact checkers like Politifact)? I'm not sure what the discrepancy is. If it truly is the case that Republicans are making more false claims than Democrats, and Politifact has an approximately equal number of false claims for each party, would this not be an indication of party-related selection bias? Politifact would be trying, possibly with some kind of "fairness doctrine" to make Republicans look more honest than they truly are. As Bill Adair noted, "we choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." No mechanisms are in place to ensure some kind of contrived fairness when one political party is more likely than the other to play fast and loose with the truth.

"Nor does PolitiFact claim on its site to present a 'fair and balanced' selection of statements, or that the statements rated are representative of the general truthfulness of the nation's political parties or the elected officials involved. "

This is correct. However, as the Washington Post Glen Kessler pointed out in a recent interview with NPR "I don't really keep track of, you know, how many Democrats or how many Republicans I'm looking at until, you know, at the end of the year, I count it up." There is no indication Politifact does any different. So it may not be the case that Politifact highlights any explicit methods for ensuring fairness (contrived or not). However, as professional journalists, is it possible they employed standard journalistic practices of objectivity so common-sense they didn't see the need to mention them?

It is possible that, given the tendency for journalists to be themselves personally liberal, that we should expect party-related selection bias as a default? It is a possibility (like many others). However, unless the entire staff is liberal (a claim that would require evidence), one has to wonder if even a few conservative journalists may actually be enough to prevent other liberal journalists from falling prey to party-related selection bias. Remember, Politifact articles are reviewed by a panel of at least 3 editors, further decreasing the probability of party-related selection bias.

In addition, a person with that hypothesis would have to explain why ratings did not favor politicians from either party over the other in 2007. Did Democrats lie more in 2007, causing party-related selection bias to make it seem like Democrats were just as honest as Republicans? Or was Politifact just not biased back then, as opposed to now? At least one would need to be investigated before drawing any conclusions.

Overall, although Politifact does not explicitly call itself "fair and balanced" (one may actually be skeptical of news organizations that do), once cannot come to the conclusion that they aren't actually fair and balanced. This would be a classic "argument from ignorance."

"And yet... 

In defending PolitiFact's "statements by ruling" summaries - tables that combine all ratings given by PolitiFact to an individual or group - Adair explained: 

"We are really creating a tremendous database of independent journalism that's assessing these things, and it's valuable for people to see how often is President Obama right and how often was Senator McCain right. I think of it as like the back of a baseball card. You know - that it's sort of someone's career statistics. You know - it's sort of what's their batting average." (C-SPAN Washington Journal, August 4, 2009) "

I will admit Bill Adair does oversell the importance of Politifact's summaries and tables. Since the selected statements are not truly random, only so much can be extrapolated from them. It may be a good idea for Politifact to provide a disclaimer stating that statements from individual politicians are not randomly chosen. They are instead chosen via the criteria outlined in their "Principles." However, with no evidence that party-related selection bias is a factor in their decision making, there would be no reason to post any disclaimer to suggest it is a factor.

HOW FAR SHOULD THE MEDIA GO TO APPEAR UNBIASED?

" Adair is also on record for lamenting the media's kneejerk inclination to treat both sides of an issue equally, particularly when one side has the facts wrong. 

In an interview with the New York Times in April 2010, Adair said: 

"The media in general has shied away from fact checking to a large extent because of fears that we'd be called biased, and also because I think it's hard journalism. It's a lot easier to give the on-the-one-hand, on-the-other-hand kind of journalism and leave it to readers to sort it out. But that isn't good enough these days. The information age has made things so chaotic, I think it's our obligation in the mainstream media to help people sort out what's true and what's not." "

Bill Adair makes a good point. Why should the media present both sides equally when one side is clearly wrong? In addition, why should the media present both parties as being equally truthful when one is actually playing fast and loose with the truth? It would seem odd that a political party that once lambasted the Fairness Doctrine would have any problem with what Bill Adair is saying.

SHIFTING THE BURDEN OF PROOF

" The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case. "

This is an interesting quote. My original critique of this quote may have been a bit over-simplified¹. As a result I thought it would be better to do a bit more thorough job on this quote, due to the implications it contains:

Notice how his last statement claims that Politifact frames Republicans in a way that suggests they lie more than Democrats. Before judging the validity of the rest of this statement, it would probably be a good idea to see if Politifact even does this at all. First off, notice that Politifact has never had an overall Dem vs GOP report card. Ostermeier was the one who did this. So this means that certain conclusions one may be tempted to draw from Ostermeier's study are not ones people may necessarily draw from Politifact itself. This is because all Ostermeier did was total up all statements made by Democrats and Republicans, and compared them. Of course he could have easily done the same with any arbitrary breakdown of statements: Men versus women, majorities versus minorities, southerners versus northerners, etc... If we found similar disparities in the data to the Dem vs GOP data, would we be accusing them of covering "political discourse with a frame that suggests this is the case?" The ultimate problem is that, Politifact rates individuals (and specific groups), not overall groupings of those individuals. In fact, Adair's earlier statement about his report cards shows that he is really only focusing on individuals. If one were to infer any trend about an entire grouping of those individuals, not only would they would need a sufficient sample size of individuals within those groups, but they would also need to filter out individuals with only a few statements to their name. It should be clear to anyone with common sense that, when you only have a small number of statements from an individual, you cannot come to any conclusion about the honesty of that individual with any decent level of certainty. As a result, people with only a few grades to their names would have to be ignored. For example, if you eliminated candidates with fewer than 5 statements to their name in 2010, you would have to eliminate Jim DeMint, Kevin McCarthy, Mike Prendergast, Dan Coats, and whole host of others... This will probably leave you with a very small sample of individuals. Seeing as how there are literally thousands of Republican politicians at all levels of government, you cannot come up with a half-way reasonable level of certainty over the honesty of Republican politicians as a whole from just Politifact's ratings alone. These kinds of things should be pretty intuitive to people just looking at Politifact's articles and report cards. As a result, Politifact can hardly be blamed for creating a framework suggesting Republicans lie more when they have never actually done such a thing, unlike Ostermeier. So as a result, does Politifact have the burden to give Republicans a convincing argument that "Republicans in fact do lie much more than Democrats?" At the very least, this is a straw man since Politifact never even implicitly made that claim. It was an observation based on a context-ignoring analysis performed by Ostermeier himself.

But let's say Politifact did look at a large number of claims from a large number of politicians to where you can gauge the honesty of enough individuals from each party to come up with a conclusion that one party does in fact lie more than the other. If Politifact is merely presenting the data they found, without coming to any conclusion, does Politifact really have the burden of proof? Whatever conclusions that are made about the data must be supported by the person making the conclusion. So if we are stuck in some dichotomy where either Politifact is biased, or Republicans lie more, should Politifact be the one with the burden of proof? As we can see, the data is agnostic to either claim. So if Politifact is making no claims, yet someone wants to claim they are biased, is it Politifact's burden to prove that Republicans lie more to keep these people from being justified in their belief in Politifact's bias? As Ostermeier points out, it is not a question over whether or not Politifact can prove outright that they don't have "ulterior motives." However, if Politifact fails to answer either of Ostermeier's questions, what can be their conclusion? Ostermeier gives Republicans no questions of their own to answer before coming to a conclusion. And as Ostermeier's last paragraph shows, he is comfortable supporting one particular conclusion over any others:


OSTERMEIER CONCLUDES BIAS

" In his August 2009 C-SPAN interview, Adair explained how the Pants on Fire rating was the site's most popular feature, and the rationale for its inclusion on the Truth-O-Meter scale: 

"We don't take this stuff too seriously. It's politics, but it's a sport too." 

By levying 23 Pants on Fire ratings to Republicans over the past year compared to just 4 to Democrats, it appears the sport of choice is game hunting - and the game is elephants. " (emphasis mine)

Up until this point, Ostermeier has been pretty good at not owning the conclusion that his data suggests Democrat-favored selection bias at Politifact. He has appeared sympathetic to concerns of this sort of bias by specifically highlighting them and not investigating other possible explanations. However, this last paragraph indicates he does actually believe that there is some kind of bias of this sort at Politifact. However, his logic in regards to "Pants on Fire" ratings suffers from the same problems readers may have when concluding in favor of any kind of party-related selection bias at Politifact (see "Conclusion" below). He fails to investigate other possible causes, instead flipping the burden of proof on Politifact to demonstrate it is not in fact guilty. In fact, if he were to look back at 2007, he would see a near polar opposite trend of assignments for "Pants on Fire" ratings. Democrats were the recipients of over half the "Pants on Fire" ratings in 2007. In addition, they received twice as many "Pants on Fire" ratings as Republicans that year, despite the fact that Democrats actually had slightly fewer total statements rated. Now there weren't a lot of "Pants on Fire" ratings that year, but it is clear the trend from 2010 was absolutely non-existent in 2007. Shouldn't that make Ostermeier wonder what changed in 2010? Was it Politifact or the Republican Party? To my knowledge no evidence has been given for the former (or even suggested), yet the rise of the TEA Party in 2010 shows the latter is certainly true.

CONCLUSION

Although it is hard to draw too many conclusions from Ostermeier's data sets. A few questions still need to be investigated before coming to any conclusions (some of these come from a previous post):

•  Could it be that many of these statements came from a single issue (or small collection of issues) where Republicans may have been more likely to spread falsehoods? A month-by-month breakdown could be useful in this regard. This kind of breakdown could also help investigate the question as to whether the Republican primaries had anything to do with the trends in 2010, as I pointed out earlier.
• Could it be that the perceived rise of anti-intellectualism in the Republican Party has made it less likely to care about factual accuracy? 
• Could the Republican party be more ideological and less prone to checking facts that could challenge their ideology? 
• Could it be that the Republican tendency to call neutral fact-checking sites "biased" creates a tendency to play fast and loose with the truth, knowing their constituents already distrust any site that could shed light on their claims? 

These are all possibilities (although I am not necessarily claiming any of them to be true). So did the article control for or check any of these, or alert readers to these possibilities? NOPE.²

Throughout most of this article, Ostermeier presents interesting data for Politifact in 2010. Although there are many possible explanations of the data, Ostermeier only really focuses on one. In addition, he ignores at least one other for no better reason than he should have for ignoring the one he focuses on as well. Ostermeier fails to ask many questions, such as whether or not the trends he found in 2010 even existed in previous years. A quick analysis of Politifact in 2007 showed this is not the case for at least one year. In the end, Ostermeier makes it clear he thinks Politifact is guilty of party-related selection bias, succumbing to the same fallacies any reader would if he/she came to the same conclusion from the information presented in this article alone (not that other articles I've seen do any better).

Overall, this article demonstrates a particularly erroneous way of detecting bias. A mere look at ratings totals cannot give a reader much information about bias. Even if one were to tally up all of Politifact's articles, few conclusions could be reached. Out of Politifact's 4 1/2 years in fact checking, 3 have been under a Democrat president, and 3 1/2 under a Democrat controlled congress. With the possibility of the party out of power being more prone to spreading falsehoods, one cannot instantly jump to conclusions if Republicans end up having more bad ratings than Democrats. I admit that demonstrating bias can be very tough. However, if someone is having THAT much trouble demonstrating bias, maybe they should wonder if they are even justified in suspecting bias in the first place. If Ostermeier had done this, maybe this article would have been more useful. Instead this article is just another failed attempt to try and poison the well for Politifact.

¹ - Update 1-31-12: Original critique: "Why should the organization have to give a convincing argument that Republicans lie more than Democrats in order to convince Republicans they do not have ulterior motives? Essentially this is the argument that, unless Politifact gives evidence it does not has ulterior motives, Republicans are justified in believing it does. This again, is a textbook argument from ignorance (see link above). Shouldn't Republicans have to ask for somewhat definitive evidence that Politifact in fact does have ulterior motives before accepting that claim as true? In fact, couldn't the data found in this entire article point to Republicans politicians spreading more falsehoods than Democrat politicians in 2010 just as much as it could point to Politifact having ulterior motives? In fact, the data from 2007 significantly challenges the latter while having practically no effect on the former."

² - Update 1-31-12: I took this bulleted list from the "SHIFTING THE BURDEN OF PROOF" section and placed it in the conclusion, to help with the revised flow of the critique.

Supposed Politifact Bias In 2010 Non-Existent In 2007

In a recent post where I critiqued a list of issues that should "keep discerning readers from trusting Politifact", I was alerted to an article examining potential selection bias in how Politifact chooses which statements to rate. The article, entitled "Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats," was written by Eric Ostermeier, "Research Associate at the Humphrey School's Center for the Study of Politics and Governance." This article has also been cited in a recent attack on fact checkers by The Weekly Standard (3-post critique: 1,2,3). Given the attention this article received, I thought it would be useful to see if this article actually gives any good reason to justify suspicion of selection bias at Politifact.

First off, it should be noted that selection bias occurs whenever a sample, taken for statistical analysis, has not been chosen randomly. As any reader can plainly see, this article does not use a random sample of statements from Politifact. Ostermeier chose statements from January 2010 through January 2011. If these dates were truly arbitrary, then it would not matter that these samples were not chosen randomly. However, as I intend to show in this post, choosing different dates can lead to remarkably different results. I will do a point-by-point critique of Ostermeier's article in another post. However, I thought I would look to see if choosing different dates made a difference to Ostermeier's results.


IS IT APPROPRIATE TO DRAW CONCLUSIONS FROM 2010 ALONE?

It is interesting to read Ostermeier's findings and see just how badly Republican officeholders are favored in the lower ratings categories during 2010 (note that when I say 2010 in this post, I'm including January 2011 to reflect Ostermeier's article). However, remember what I said earlier about how these findings are not based on a truly random sample. This should beg the question over whether this trend has existed in previous years. I decided to do a quick analysis of the 8 months of 2007 when Politifact first started rating statements from politicians. I chose this time period because it was relatively short and easy to analyze. My results turned out to be quite a bit different from Ostermeier's:

 Let's compare these results to the findings of Ostermeier for 2010:

•  Out of 39 statements rated "False" or "Pants on Fire," each main party received approximately 50% of the rulings. This is much different from 2010, where Republicans received 75.5% of these ratings.
• Politifact has devoted approximately equal time between Republicans (124, 52%)  and Democrats (110, 46%). This is nearly the same as 2010.
• Republicans were graded in the "False" or "Pants on Fire" categories 16% of the time (39% in 2010) while Democrats were rated in these categories 16% of the time as well (12% in 2010). This absolutely nowhere close to the "super-majority" found in 2010.
• 73% of Republican statements in 2007 received one of the top 3 ratings (47% in 2010) as opposed to 75% of Democratic statements (75% in 2010). When looking at "True" statements alone, Republicans beat Democrats with 5% more of their statements receiving the coveted award than Democrats. It is hard to see how one party was considered more or less truthful than the others. It depends on how much credit you give the "Half True" rating, as opposed to the "True" rating.
• Republicans received a slightly larger percentage of "Mostly False" ratings than Democrats (1.79%). This is the same as in 2010. However, this only results in a 2% difference between Republicans and Democrats for the bottom 3 ratings. This is MUCH different from the 28% difference in 2010. 

As you can see, the results from 2007 can seriously undermine many possible conclusions that a person could draw from the 2010 data. The fact that the kind of results you see are dependent on the dates chosen show this is not a random sample. In fact, focusing solely on 2010 to hint at the possibility of Democrat-centered selection bias within Politifact would actually be an great example of cherry-picking data, a well known fallacy.

This is a great example of why every reader should remember his/her critical thinking cap when analyzing statistics. Don't leave home without it!

Politifact Bias (or not) [Defending the Truth-O-Meter]

The Blog "Politifact Bias" seems to be a pretty popular blog for conservatives who want to confirm their poorly-justified belief that the neutral fact checking site known as Politifact is actually left-biased (at least according to Google Search Results). I have yet to look at this blog directly since they rarely publish original posts. However, I thought I may as well check out their "PolitiFact 2011: A review" post to get an idea of what they considered to be significant evidence of the diagnosed liberal bias of Politifact in 2011.¹ As far as I can tell, this blog merely collects criticisms of Politifact they find around the internet. In my experience, these criticisms can be extremely poor. They listed "a rundown of the issues that should keep discerning readers from trusting PolitiFact:"

1)  PolitiFact persistently ignores the effects of selection bias.  It simply isn't plausible that editors who are very probably predominantly liberal will choose stories of interest on a neutral basis without some systematic check on ideological bias.  PolitiFact, for example, continues to publish candidate report cards as though selection bias has no effect on the report card data.

Basically, the linked article⁴ is a great example for a subject I was planning on writing about, entitled "The Wrong Ways To Detect Bias." One of the methods I was planning on talking about was exactly what these people did. They just compiled a list of articles on Politifact over the course of a year and came to the conclusion that, since the worst ratings tend to favor republicans more than democrats, they must be biased in their selection in favor of the left. The problem with this is that their conclusion simply does not follow. Their are other possible reasons for this kind of result that have nothing specifically to do with Politifact being biased.

1. To give them the most credit, it is possible there may be some liberal selection bias at work here, but not in the way they think. Politifact often gets ideas for statements to check from their readers. It is possible most of their readers are liberal, which souds quite plausible given the fact that conservatives seem so ready to accuse Politifact of bias. This is no fault of Politifact, nor does it show any bias on their part. The fault lies squarely with Republicans who chose not to submit statements to Politifact and instead waste their time complaining about perceived bias (a vicious cycle). It should also be noted that the definition of Selection Bias is "A type of bias caused by choosing non-random data for statistical analysis." Well duh! The article itself mentions that Politifact Bill Adair noted "We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." That right there should show that there is selection bias, though not liberal selection bias. However, the article intends to show that the bias is toward the left, instead of toward important statements. If the article was intended to merely accuse Politifact of selection bias, they would basically be attacking a straw man since the main goal of Politifact is not to tally true and false statements to figure out which candidates are more honest (although this does happen occasionally). It should also be noted the selection of Politifact articles used in this calculation are very much non-random. As the article mentioned, the time period selected was when Democrats were in power. This could have easily affected the results, as I will get into later.


2. Could it be that Republicans were telling falsehoods more often during that time period? The article does mention this. "One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site. However, there is no evidence offered by PolitiFact that this is their calculus in decision-making." Yet they don't look for evidence that their explanation is the better one. To show the fallacy in this kind of thinking, let me present an analogy: Let's say i have two dogs that roam free in my house all day, Rex and Dex. I come home one day to find my couch clawed to pieces. I cannot find evidence that Rex did it, so I assume Dex did it (even though I found no evidence specifically pointing to Dex). This is exactly the same fallacy they employ. They don't specifically have evidence of a Republican tendency towards falsehoods so they assume it must be a Politifact tendency towards liberal bias. Like I noted earlier, the article did not randomly chose articles. Could it be that, since the Republican Party was out of power, they had a greater tendency towards falsehoods? Could it be that the ratings success of Fox News created a podium for more Republican falsehoods? Could it be that, with the rise of the right-wing Tea Party, extreme views have tended to produce more falsehoods? Could it be that the possible rise of anti-intellectualism in the Republican Party has made it more likely to back falsehoods? Could it be that the Republican Party itself is more prone to being dishonest? Could the Republican party be more ideological and less prone to checking facts that could challenge their ideology? Could it be that the Republican tendency to call neutral fact-checking sites "biased" creates a tendency to play fast and loose with the truth? These are all possibilities (although I am not necessarily claiming any of them to be true). So did the article control for or check any of these? NOPE. They could have checked other neutral fact-checking sites to see if they came to similar conclusions as Politifact. They could have compared these results with articles from before Obama was elected. They just dismissed these hypotheses for no better reason than they would have for dismissing their own hypothesis.


3. It could be that the significance of the truer statements was much higher than that of the false statements. Not all statements are equal. Chris Christie's statement about the city of Newark's graduation rate has far less of an impact on party image than the Democratic claims about Republicans trying to end Medicare. An overall look at a large number of graded statements can give you a decent look at a candidate's honesty, but you need to actually check the statements to see how significant they are. If Politifact is giving Democrats more high-profile false's then it doesn't matter that there are fewer. The primary focus of Politifact once again is to read the about ratings they give these statements. So a handful of high profile falsehoods for Democrats can look much worse than a bunch of inconsequential falsehoods for Republicans.

It should be pretty clear that merely tallying Politifact ratings doesn't give you anything remotely conclusive about the theoretical bias of a site.

2)  PolitiFact continues to publish obviously non-objective stories without employing the journalistic custom of using labels like "commentary," "opinion" or even "news analysis."  Readers are implicitly led to believe that stories like an editorial "Lie of the Year" selection are objective news stories.

Any evidence for this? A link or two maybe? If they wrote an article about this, it would be useful at this point. I may search for an article in their archives and come back to this later. As it stands, this is an unsubstantiated statement. I will, however, agree that the "Lie of The Year" title is a bad title since there is a difference between people who lie (know their statements are false) and people who are careless with the truth (don't necessarily know their statements are false). However, Politifact does have some leverage here since the 2009, 2010, and 2011 Lies of the Year have been tackled by Politifact, as well as other Fact Checking sites, so often, it is hard to imagine these politicians do not know their claims have been debunked. It is also good to note that Politifact lets readers know the "Lie of The Year" is just a consensus opinion of their editors and reporters, not objective news. So readers are actually not led to believe the Lie of The Year is an objective news story.

3)  PolitiFact continues to routinely apply its principles of analysis unevenly, as with its interpretation of job creation claims (are the claims assumed to refer to gross job creation or net job creation?).

The link provided goes to a re-post of this exact same blog entry. I'm interested to see what they are talking about. If I find evidence I will post my critique.³

4)  PolitiFact has yet to shake its penchant for snark.  Snark has no place in objective reporting (see #2 above).  Unfortunately, PolitiFact treats it like a selling point instead of a weakness, and PolitiFact's intentional use of it has apparently influenced Annenberg Fact Check to follow suit.

I'm not 100% sure what he means by "Snark"² However, I will assume he means "Use of sarcasm or malice in speech." I'm not sure I have ever seen sarcasm. But I fail to see the problem here. Politifact is in the business of spreading awareness of "Fact Checking." It helps keep readers interested and coming back for more (although it's nowhere near as common as in Glenn Kessler's articles). Overall, it has no negative effects on the actual research. In fact, it can help get readers fired up about falsehoods and dishonesty. Other than personal taste, I see no problem with this. In fact, any case of "snark" I have ever seen has been quite subtle, meaning I cannot see it honestly turning away any readers that care more about reality than ideology.

It seems as though none of these "issues" should "keep discerning readers from trusting Politifact" so long as they care about quality fact-checking to help ensure they don't subscribe to false beliefs. Only one of these "issues" even had a linked article, and the content of that linked article came nowhere near demonstrating bias at Politifact. I will empathize that showing the existence of bias is no easy task (usually). But if you are having THAT much trouble exposing bias, maybe you should wonder if the bias really exists at all...


¹ - Update 1-10-12: The writers of "Politifact Bias" have informed me that this post was meant to be just a summary, and was not meant to give significant evidence of bias. However, this does not excuse the fact that they did list "the issues that should keep discerning readers from trusting PolitiFact," without providing evidence for a few of them. This does not change the rest of this post because all this post does is check to see if these "issues" significantly exist at all.

² - Update 1-10-12: The writers of "Politifact Bias" have informed me that, by snark, they mean that Politifact is conveying their own biases. When I asked for an example, they provided me with a quote from a July 2011 politifact article talking about Newt Gingrich:"With the 2012 election in their sights, Republican candidates spend most of their time trying to prove that President Barack Obama and the Democrats will make the economy worse." However this is a terrible example given that this is not an uncontroversial fact. There is no hint of the reporter's feelings, biases, or prejudices. Nor is there any hint of unnecessary partisan adjectives. It sounds as if this example fails to substantiate their claim that Politifact is guilty of "snark." 

Update 1-10-12: I have edited this post to enhance readability. No significant changes were made to the content.
 
³ Update 1-10-12: A writer of "Politifact Bias" has informed me that the link was to a keyword search on one of his blogs (seems a bit unnecessarily distracting and cryptic). He verified a certain post did a good job underscoring his complaint. In this post he mostly just lists each case and states whether it was helped or harmed by the inclusion/exclusion of "gross jobs." The problem is he mostly fails to investigate context. There are some statements where it is appropriate to look at net jobs, such as a statement that looks at "jobs added" as a whole. Conversely, there are statements where it is appropriate to ignore lost jobs. When a politician states the stimulus did not create one job, it doesn't matter how many jobs were already lost as a result of the recession, just so long as at least one job was directly created by the stimulus. Since the mechanism for lost jobs and the mechanisms for gained jobs are different, it is more than appropriate to ignore what has nothing to do with the mechanism under question. For example, If I went to a village under attack by militants and pulled a single child to safety, thereby saving it's life, should I not be credited with saving that kid's life just because more than one life was lost in that village from the attack? Any reasonable person would find this absurd. Granted he attempted an investigation into the context of choosing based on the word "created." I will give him some credit for that, but he did not investigate whether or not Politifact was consistent when the mechanism that "created" jobs was the same as the mechanism that caused jobs to be lost. He just simply focused on the word "created." As the stimulus example shows, his list definitely ignores important context. It should also be noted that Politifact has actually made the context used in their decision making well known (see last link).

⁴ - Update 1-16-12: I have also completed a comprehensive critique of this article as well.