Tuesday, January 10, 2012

The Weekly Standard Attacks Fact Checkers (Part 2)


This is part 2 of a 3-part series entitled "The Weekly Standard Attacks Fact Checkers." Part 1 can be read here.

DEFENDING THE TRUTH-O-METER

The AP's Fact Check was not the only target of this article. What is a good attack on the practice of Fact Checking without a good shot at Politifact?
Here’s a not-atypical case study. On November 7, 2010, newly elected Senator Rand Paul appeared on ABC’s This Week with Christiane Amanpour. One of the topics of discussion was pay for federal workers. “The average federal employee makes $120,000 a year,” Paul said. “The average private employee makes $60,000 a year.” Given that the news these days often boils down to debates over byzantine policy details, Paul’s statement is about as close to an empirically verifiable fact as you’re likely to hear a politician utter. And the numbers are reasonably clear. According to the latest data from the Bureau of Economic Analysis​—​yes, that’s a government agency​—​federal workers earned average pay and benefits of $123,049 in 2009 while private workers made on average $61,051 in total compensation...
(section removed due to lack of relevance)
Not only is what Senator Paul said about federal pay verifiably true, his simple recitation of the most basic facts of the matter doesn’t even begin to illustrate the extent of the problem. Yet PolitiFact rated Senator Paul’s statement “false.” According to PolitiFact’s editors, because Paul did not explicitly say the figures he was citing include pay and benefits, he was being misleading. The average reader would assume he was only talking about salary. “BEA found that federal civilian employees earned $81,258 in salary, compared to $50,464 for private-sector workers. That cuts the federal pay advantage almost exactly in half, to nearly $31,000,” writes PolitiFact. So the average federal employee makes a mere $31,000 more a year in salary than the average private sector worker​—​but also gets a benefits package worth four times what the average private sector worker gets. PolitiFact further muddies the waters by suggesting that the discrepancy between public and private sector averages isn’t an apples-to-apples comparison. Again, Andrew Biggs, the former Social Security Administration deputy commissioner for policy, and Jason Richwine of the Center for Data Analysis, writing in these pages (“Yes, They’re Overpaid: The Truth About Federal Workers’ Compensation,” February 14, 2011), observed that the most favorable studies of federal worker compensation “controlling for age, education, experience, race, gender, marital status, immigration status, state of residence, and so on” still find federal workers are overpaid by as much as 22 percent.
At the very least, this criticism underscores the principle that most falsehoods have a small grain of truth. However, Politifact has to give Rand Paul's statement a rating based on the expected interpretation of a reasonable person. Since he included both pay and benefits, without specifying so, this alone can arguably justify a bad rating as a viewer could easily overestimate the amount of money a federal worker makes. In addition, Rand Paul makes it seem as though federal pay is DOUBLE private sector pay. Politifact notes that, once the comparison is apples-to-apples, that number is closer to $7000, or a little more than 10% greater than comparable private sector jobs. How does Hemingway respond? He retorts that "the most favorable studies" (if you read the article, you will notice that all these studies are either by conservative think tanks or cherry picked government jobs) actually put that number as low as 22%. However, 22% is still a far cry from 100%. So, even by Hemingway's standards, the actual differences are much less than what Rand Paul cites. How does quadrupling the difference in pay not warrant a major downgrade in rating?

THE AP IS NOT OFF THE HOOK YET

Hemingway then returns to his attacks on the AP's fact checking operations. During the nomination of Justice Elena Kagan, the AP ran a Fact Check article dealing with claims that she was anti-military. Hemingway decided the AP was fact checking opinion and not actual facts.
Again, here are the facts: Kagan was a dean at a law school that had banned ROTC over what she referred to as the military’s “repugnant” ban on openly gay service. This was, not surprisingly, an issue raised when she was nominated for her current position on the Supreme Court. The AP’s own fact check even noted that she filed a legal brief in support of colleges that wanted to uphold their policies restricting military recruiters on campus, though she opted not to join the lawsuit. Whether the fact that Kagan valued making a statement about gay rights over supporting the vital national security effort of military recruitment amounts to being “antimilitary” is quite obviously a matter of opinion, as is the charge that she’s an “ivory tower peacenik.”
Actually the prefix "anti" literally means "one that is opposed." So it stands to reason that "anti-military" means "one that is opposed to the military." This is a factual claim, not an opinion. It is also demonstrably false. The AP points out: "In the heat of the recruitment debate, and after, Kagan praised military service as "the noblest of all professions" and a "socially valuable career path" that should be open to all. "I know how much my security and freedom and indeed everything else I value depend on all of you," she told West Point cadets." She even sent out an email defending "the school's earlier decision to set aside restrictions on the armed forces and to allow - not ban - military recruitment. (emphasis mine)." Maybe Hemingway can explain  how supporting the military can possibly be consistent with being anti-military.

It was actually Newt Gingrich who made the claim on "Fox News Sunday." Fact checkers have to evaluate statements based on how a reasonable person would interpret them. There should be no argument that someone who heard Newt Gingrich's statement would think Elena Kagan is actually against the military, based on his statement alone. Furthermore,  "Whether the fact that Kagan valued making a statement about gay rights over supporting the vital national security effort of military recruitment amounts to being “antimilitary” is quite obviously a matter of opinion" is not a standard I'd venture to think Hemingway would follow consistently. If Democrats were to say that Republicans are utterly opposed to fiscal responsibility because they favor avoiding new taxes over balancing the budget, I doubt Hemingway would treat such a claim as merely opinion.
Revealingly, the inflammatory phrase “ivory tower peacenik” was never actually used by Kagan’s critics​—​it was from the AP headline and the first sentence of its fact check: “Elena Kagan is no ivory-tower peacenik.” Here the AP pulled off a seriously impressive feat of yellow journalism. By caricaturing the tone of the actual criticisms, the AP set up a straw man for its “fact check” to knock down before the reader even got past the headline.
It should be noted that the AP never claimed Republicans called Kagan an "ivory-tower peacenik." In order for this to actually be a straw man, the AP would have had to criticize this characterization of Justice Kagan. In fact, nowhere in the article did the AP do that. The irony here is that Hemingway himself essentially criticized a straw man instead of what the AP actually wrote. Hemingway should also know "ivory-tower Peacenik" is merely a catchy title. Seeing as how nowhere in this article does Hemingway actually diagnose fact checkers as liars, as his title suggests, the question arises over whether or not he is, by his own standards, guilty of yellow journalism. Since I doubt he would stoop to this level of hypocrisy, I doubt he could fairly justify the claim the AP is itself guilty of yellow journalism.

The Weekly Standard's attack on the AP Fact Checking operation continued:
At the most basic level, the media’s new “fact checkers” remain obdurately unwilling to let opinions simply be opinions. Earlier this year the AP fact checked a column by former GOP presidential candidate Tim Pawlenty in which the former Minnesota governor asserted that “Obamacare is unconstitutional.” Contra Pawlenty, the AP intoned, “Obama’s health care overhaul might be unconstitutional in Pawlenty’s opinion, but it is not in fact unless the Supreme Court says so.” The AP aligns itself here with the myth of judicial supremacy, namely the mistaken idea that the Supreme Court has a monopoly on deciding what is and is not constitutional. But aside from this amateur-hour excursion into legal theory, the AP betrays a more basic problem of reading comprehension: Pawlenty’s USA Today column appeared in a section of the newspaper clearly labeled OPINION in large, bold letters.
Hemingway has completely confused me. The AP clearly admitted that Pawlenty was merely expressing his opinion. So why is Hemingway accusing them of faulty reading comprehension? Did Hemingway forget the quote he JUST posted in the preceding paragraph?

As to the issue of the "myth of judicial supremacy," The AP may be treating the Supreme Court as a consensus opinion with the final say over whether or not Obamacare is Constitutional. In effect, they may be merely reminding people that Obamacare has not been ruled upon by the Supreme Court so it should not be considered a legal fact that Obamacare is unconstitutional, just personal opinion. I will admit I personally do not like the way the AP worded the phrase. It is definitely over-simplified, but Hemingway has failed to justify calling it "amateur-hour excusion into legal theory."

In addition, it should also be noted that when a person makes a statement that can be reasonably verified as true or false, it ceases to just be an opinion, but an actual FACT. And just because a claim appears on the Opinion page does not mean it is just opinion. It is common for writers to present facts on the Opinion page to help justify their opinions. If I said the earth was flat on an Opinion page, does that make my claim any less false? As I have noted before, Fact Checkers have to assess these claims based on how they would expect a reasonable person would interpret the claims. As a result, the AP provided information enough to cover many reasonable interpretations.

“The AP also did an extensive investigation into Obama’s handling of the Gulf spill, and concluded it ‘shows little resemblance to Katrina,’ ” writes Sargent. “As [liberal Washington Monthly blogger] Steve Benen noted in lauding this effort, the AP definitively debunked a key media narrative as ‘baseless.’ ” One could ask whether the BP oil spill was being compared with Katrina simply because of its relative proximity and public opinion that the Obama administration handled the crisis similarly poorly. But why bother? The very idea of fact checking a broad comparison should send readers who give a damn about facts screaming for the exits.

Unlike checking individual facts, broad comparisons create a much larger task for a fact checker. Although one could theoretically find a near unlimited set of potential comparisons, the overwhelming majority of these will likely be trivial. As a result, many broad comparisons can be reduced to a small manageable set of significant comparable facts. When this is the case, broad comparisons are actually quite useful. For instance, it is possible to compare significant aspects of Romneycare and Obamacare to determine just how similar those bills actually are. It is also possible to determine if comparisons of a given politician to Hitler are justified. To be fair, some broad comparisons cannot be reduced to a manageable set of significant check-able facts. However a reader who runs "screaming for the exits" just because they are confronted with a broad comparison has likely fallen into the fallacy that all comparisons not comprised of a single fact must be incomparably large. Such a fallacy has no place in the practice of Fact Checkers.



To be continued in part 3 of "The Weekly Standard Attacks Fact Checkers"

4 comments:

  1. Hello, Knoxville, I am commenting here because I don't want to be bothered by Bryan White. Just want you to know that I've been accumulating everything relative to PolitiFact TruthoMeter ratings in Excel since 2010, although with not quite the depth of analysis that you have. You've given me some good ideas, and if it is helpful to you here is my blog--http://politi-psychotics.blogspot.com/search/label/Truth%20Index I would appreciate your suggestions and comments. There's also a guy named Steve at http://quibblingpotatoes.blogspot.com who has done some worthwhile analysis of PF Truthometer stats. Thanks, Karen

    ReplyDelete
  2. Yeah one of my other posts dealt with a post on Bryan's blog (I also have still yet to post). I posted the critique as a comment on one of his posts and he started a long process of trolling my blog, specifically the post regarding his (understandably) and two posts pointing out problems with concluding the existence of selection bias in Ostermeier's study.
    I'll take a look at yours and Steve's blogs and give a few pointers if I can think of any.
    I will note that one issue I have with statistical and population analyses about politifact is that they can be misleading, when proper controls are not met (which can be very tough to meet). If you look at my post over Ostermieir's article, I show that his results neither support selection bias, nor a Republican tendency to lie more often.

    ReplyDelete
  3. I took a look at your guys' sites and they look pretty good on numbers. Plus I like that you guys don't seem to really jump to conclusions often. However, I will suggest one really important point I made on a previous post about the question over Politifact's ratings meaning republicans lie more or if there is selection bias:

    "First off, notice that Politifact has never had an overall Dem vs GOP report card. Ostermeier was the one who did this. So this means that certain conclusions one may be tempted to draw from Ostermeier's study are not ones people may necessarily draw from Politifact itself. This is because all Ostermeier did was total up all statements made by Democrats and Republicans, and compared them. Of course he could have easily done the same with any arbitrary breakdown of statements: Men versus women, majorities versus minorities, southerners versus northerners, etc... If we found similar disparities in the data to the Dem vs GOP data, would we be accusing them of covering "political discourse with a frame that suggests this is the case?" The ultimate problem is that, Politifact rates individuals (and specific groups), not overall groupings of those individuals. In fact, Adair's earlier statement about his report cards shows that he is really only focusing on individuals. If one were to infer any trend about an entire grouping of those individuals, not only would they would need a sufficient sample size of individuals within those groups, but they would also need to filter out individuals with only a few statements to their name. It should be clear to anyone with common sense that, when you only have a small number of statements from an individual, you cannot come to any conclusion about the honesty of that individual with any decent level of certainty. As a result, people with only a few grades to their names would have to be ignored. For example, if you eliminated candidates with fewer than 5 statements to their name in 2010, you would have to eliminate Jim DeMint, Kevin McCarthy, Mike Prendergast, Dan Coats, and whole host of others... This will probably leave you with a very small sample of individuals. Seeing as how there are literally thousands of Republican politicians at all levels of government, you cannot come up with a half-way reasonable level of certainty over the honesty of Republican politicians as a whole from just Politifact's ratings alone. "

    http://contentinreality.blogspot.com/2012/01/politifact-selection-bias-or-bad.html

    ReplyDelete
  4. I tend to be pretty busy most of the time so I do not have too much time to give many pointers (or get in lengthy comment wars with people like Bryan). However, most of what I could point out is listed in the couple of articles I wrote on this blog about criticisms of fact checkers, plus possibly a few I may point out in future criticisms. I do not seem to have near the experience you do though in dealing with these critics so I'm not sure how many useful pointers I have, lol.

    ReplyDelete