Sunday, January 15, 2012

Supposed Politifact Bias In 2010 Non-Existent In 2007


In a recent post where I critiqued a list of issues that should "keep discerning readers from trusting Politifact", I was alerted to an article examining potential selection bias in how Politifact chooses which statements to rate. The article, entitled "Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats," was written by Eric Ostermeier, "Research Associate at the Humphrey School's Center for the Study of Politics and Governance." This article has also been cited in a recent attack on fact checkers by The Weekly Standard (3-post critique: 1,2,3). Given the attention this article received, I thought it would be useful to see if this article actually gives any good reason to justify suspicion of selection bias at Politifact.

First off, it should be noted that selection bias occurs whenever a sample, taken for statistical analysis, has not been chosen randomly. As any reader can plainly see, this article does not use a random sample of statements from Politifact. Ostermeier chose statements from January 2010 through January 2011. If these dates were truly arbitrary, then it would not matter that these samples were not chosen randomly. However, as I intend to show in this post, choosing different dates can lead to remarkably different results. I will do a point-by-point critique of Ostermeier's article in another post. However, I thought I would look to see if choosing different dates made a difference to Ostermeier's results.


IS IT APPROPRIATE TO DRAW CONCLUSIONS FROM 2010 ALONE?

It is interesting to read Ostermeier's findings and see just how badly Republican officeholders are favored in the lower ratings categories during 2010 (note that when I say 2010 in this post, I'm including January 2011 to reflect Ostermeier's article). However, remember what I said earlier about how these findings are not based on a truly random sample. This should beg the question over whether this trend has existed in previous years. I decided to do a quick analysis of the 8 months of 2007 when Politifact first started rating statements from politicians. I chose this time period because it was relatively short and easy to analyze. My results turned out to be quite a bit different from Ostermeier's:



 Let's compare these results to the findings of Ostermeier for 2010:
  •  Out of 39 statements rated "False" or "Pants on Fire," each main party received approximately 50% of the rulings. This is much different from 2010, where Republicans received 75.5% of these ratings.
  • Politifact has devoted approximately equal time between Republicans (124, 52%)  and Democrats (110, 46%). This is nearly the same as 2010.
  • Republicans were graded in the "False" or "Pants on Fire" categories 16% of the time (39% in 2010) while Democrats were rated in these categories 16% of the time as well (12% in 2010). This absolutely nowhere close to the "super-majority" found in 2010.
  • 73% of Republican statements in 2007 received one of the top 3 ratings (47% in 2010) as opposed to 75% of Democratic statements (75% in 2010). When looking at "True" statements alone, Republicans beat Democrats with 5% more of their statements receiving the coveted award than Democrats. It is hard to see how one party was considered more or less truthful than the others. It depends on how much credit you give the "Half True" rating, as opposed to the "True" rating.
  • Republicans received a slightly larger percentage of "Mostly False" ratings than Democrats (1.79%). This is the same as in 2010. However, this only results in a 2% difference between Republicans and Democrats for the bottom 3 ratings. This is MUCH different from the 28% difference in 2010. 
As you can see, the results from 2007 can seriously undermine many possible conclusions that a person could draw from the 2010 data. The fact that the kind of results you see are dependent on the dates chosen show this is not a random sample. In fact, focusing solely on 2010 to hint at the possibility of Democrat-centered selection bias within Politifact would actually be an great example of cherry-picking data, a well known fallacy.

This is a great example of why every reader should remember his/her critical thinking cap when analyzing statistics. Don't leave home without it!

37 comments:

  1. Funny stuff. I'll be back later to explain some of your errors to you.

    ReplyDelete
  2. it should be noted that selection bias occurs whenever a sample, taken for statistical analysis, has not been chosen randomly. As any reader can plainly see, this article does not use a random sample of statements from Politifact.

    Ostermeier announced his selection criteria. He chose persons elected to public office at one time or another. His selections improved the quality of his study (since chain emails distort the findings). He used all of the applicable data from 2010. So his conclusions are good for 2010, obviously. 2007 data do not matter with respect to 2010. All of this is so obvious that it should go without saying. There's no flaw with selection bias in Ostermeier's study. Selection bias is important when one is trying to obtain a representative sample (one can't get more representative than using the entire set). A sample encompassing all the relevant data for 2010 is obviously good for conclusions regarding 2010. If you can find Ostermeier claiming that the 2010 data apply to 2007, then you've got a case.

    I'll give you a chance to respond to this before moving to the next problem.

    ReplyDelete
  3. "Ostermeier announced his selection criteria. He chose persons elected to public office at one time or another. His selections improved the quality of his study (since chain emails distort the findings). He used all of the applicable data from 2010. So his conclusions are good for 2010, obviously."
    It depends on the population he is looking at. He actually is not completely consistent over his population. Notice the title says nothing about 2010. The suggestions he investigates say nothing about 2010 in particular. They are generalizations about Politifact's selection process. In fact, the interview he gets from Bill Adair comes from 2009. If he was worried about JUST 2010, why would he bring up an interview from 2009? He clearly, throughout the entire article, talked about possible selection bias in the way Politifact selects which statements to read. This applies to all years Politifact has been around, including 2007.

    "2007 data do not matter with respect to 2010."
    2007 matters to generalizations about Politifact as whole, which the article clearly looks at.

    "All of this is so obvious that it should go without saying. There's no flaw with selection bias in Ostermeier's study. Selection bias is important when one is trying to obtain a representative sample (one can't get more representative than using the entire set)."
    It should be obvious from his generalizations about Politifact that he is talking about more than just 2010.

    "A sample encompassing all the relevant data for 2010 is obviously good for conclusions regarding 2010. If you can find Ostermeier claiming that the 2010 data apply to 2007, then you've got a case."
    By suggesting generalizations about Politifact's selection process, he is suggesting this is not just limited to 2010.



    In fact, this article is investigating whether or not there is evidence for selection bias within politifact. It would not make sense for his population to JUST be 2010.

    ReplyDelete
  4. Notice the title says nothing about 2010. The suggestions he investigates say nothing about 2010 in particular. They are generalizations about Politifact's selection process. In fact, the interview he gets from Bill Adair comes from 2009. If he was worried about JUST 2010, why would he bring up an interview from 2009?

    1) I note that you left quite a bit of the content of this post out of your title as well. That observation is immaterial.

    2) The quote from Adair comes from 2009 because you have to take PF's statement about its selection process where you can find them. Another non-issue.

    He clearly, throughout the entire article, talked about possible selection bias in the way Politifact selects which statements to read. This applies to all years Politifact has been around, including 2007.

    Yet somehow for purposes of this critique you were able to identify the suggestion of bias to the year 2010 (note your title). How did you manage that? By jumping to conclusions?

    There's such a thing as context. 2010 was the most recent calendar year when Ostermeier wrote. There's nothing unreasonable about his presentation, however much you would like to pretend otherwise. You've got nothing. You get the last word, then on to the next problem.

    ReplyDelete
  5. "1) I note that you left quite a bit of the content of this post out of your title as well. That observation is immaterial."
    This is more than just the title of article. He talks about possible conclusions about Politifact IN GENERAL later on in the article.

    "
    2) The quote from Adair comes from 2009 because you have to take PF's statement about its selection process where you can find them. Another non-issue"

    Poor excuse. If he was talking about 2010 in particular, this interview would have been irrelevant!

    "Yet somehow for purposes of this critique you were able to identify the suggestion of bias to the year 2010 (note your title). How did you manage that? By jumping to conclusions?"
    Where did I say he just suggested bias in 2010?

    "There's such a thing as context. 2010 was the most recent calendar year when Ostermeier wrote. There's nothing unreasonable about his presentation, however much you would like to pretend otherwise. You've got nothing. You get the last word, then on to the next problem."
    He clearly talks about generalizations of Politifact in general. Yes 2010 was the most recent year. But you cannot make generalizations without looking at other years, unless he specifically suggests Politifact may have not been guilty of selection bias in previous years.

    ReplyDelete
  6. I'll address one representative jaw-droppingly bizarre question, then on to the next issue.

    "'(S)omehow for purposes of this critique you were able to identify the suggestion of bias to the year 2010 (note your title). How did you manage that? By jumping to conclusions?'
    Where did I say he just suggested bias in 2010?"


    Um, in your title? Read it.

    I chose this time period because it was relatively short and easy to analyze.

    Can you think of any reasons why that time period might not be the best one to use for comparison? Such as the fact that PolitiFact during that time focused only on campaign statements? Such as the fact that 2007 represented a primary season where damage to candidates would primarily figure in intra-party election contests? Or perhaps even the fact that PolitiFact at that time was a partnership between the St. Petersburg Times and the Congressional Quarterly with writers from both operations contributing stories? The reasonable observer concludes that PolitiFact is not the operation today that it was then. Not "PolitiFact as we knew it" one might say. That partial year turns out as a statistical outlier. No pun intended.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. "Um, in your title? Read it."
    What a particularly weak interpretation. Weak interpretations create straw men. Read it carefully.

    "Can you think of any reasons why that time period might not be the best one to use for comparison? Such as the fact that PolitiFact during that time focused only on campaign statements?"
    Din Ding Ding we have a winner! That is an example of a great question to ask about the 2010 data as well. I meant to write that exact question in my analysis of the 2010 data. You may be finally getting the main point I was making in my critique of the 2010 data. He failed to ask a whole heap of questions. You just thought of one. Good Job!

    "Such as the fact that 2007 represented a primary season where damage to candidates would primarily figure in intra-party election contests?"
    Am I mistaken in remembering a heavily televised primary season where Tea Party candidates attacked the Republican establishment mercilessly... A breakdown of claims by month that year culd have helped.

    "Or perhaps even the fact that PolitiFact at that time was a partnership between the St. Petersburg Times and the Congressional Quarterly with writers from both operations contributing stories?"
    More great questions, similar to what should have been asked in the 2010 data. To your knowledge, was the staff all that different back then? The vast majority of editors and writers come from the St Petersburg/Tampa Bay Times and the CQ. In fact, it is still a project of the Tampa Bay Times and its affiliates. The Times does not own CQ anymore but many Politifact writers and editors have still stayed. Politifact has more partnerships but thats about it.

    "The reasonable observer concludes that PolitiFact is not the operation today that it was then. Not "PolitiFact as we knew it" one might say."
    It has changed a bit, but so has the Republican Party since then (TEA PARTY).

    "That partial year turns out as a statistical outlier. "
    How do you know it is a statistical outlier? I looked at 237 statements graded, ~10% of Politifact's total statements). Ostermeier looked at 379 (applicable), ~17% of total statements graded. Both encompass much more than the max 5% needed to point to a total statistical outlier. Neither are random samples of data. So how are you coming to this conclusion?

    ReplyDelete
  9. "What a particularly weak interpretation."

    lol
    Tell me the correct interpretation and I'll see if I can reconcile the two. Though if you're just being evasive you probably should totally ignore the request that you provide the accurate interpretation.

    "You just thought of one. Good Job!"

    Given your great wisdom on the subject I find it surprising you would duplicate the supposed error with such seeming ease.

    "Am I mistaken in remembering a heavily televised primary season where Tea Party candidates attacked the Republican establishment mercilessly"

    We can allow that possibility, but using that excuse is thin indeed when 2010 was only partly primary season. All of 2007 was primary season. How would you ever get apples-to-apples out of the 2007 data?

    "More great questions, similar to what should have been asked in the 2010 data."

    Your response reeks of tu quoque, and that's reading it charitably.

    Of about 40 "False" and "Pants on Fire" statement rated in 2007 under half were penned by current PF staff (Adair, Drobnic Holan, Hollyfield). The editorial boards may be at least as important, but PolitiFact does not identify those (part of their program of transparency, doubtless).

    "It has changed a bit, but so has the Republican Party since then (TEA PARTY)."

    It's nice of you to volunteer help in undercutting your attempted comparison.

    "How do you know it is a statistical outlier?"

    Have I mentioned that I'm busy doing a research project for which PolitiFact is the subject? Trust me or don't. You can do your own research on it. Either 2008 or 2009 should serve. Or go for 2011.

    ReplyDelete
  10. I wrote:"Of about 40 "False" and "Pants on Fire" statement rated in 2007 under half were penned by current PF staff (Adair, Drobnic Holan, Hollyfield)."

    Missed a Hollyfield story in my first count. Half of the 44 entries were written by current staff, if the current count is correct.

    ReplyDelete
  11. "Tell me the correct interpretation and I'll see if I can reconcile the two. Though if you're just being evasive you probably should totally ignore the request that you provide the accurate interpretation."
    It seems pretty obvious, but ill bite: The data he provides is for 2010, so at the very least he assumes bias in 2010. However, he also, as I have pointed out, assumes bias throughout Politifact's history.

    "Given your great wisdom on the subject I find it surprising you would duplicate the supposed error with such seeming ease."
    I used the same methods as him. That was the point. I used the same methods as him, but chose a different year. As I pointed out in my other article, this should spark at least a few relevant questions.

    "We can allow that possibility, but using that excuse is thin indeed when 2010 was only partly primary season. All of 2007 was primary season. How would you ever get apples-to-apples out of the 2007 data?"
    But how do you know a significant number of the bad ratings for republicans didn't come from that? That is why the question is relevant in 2010. AS you pointed out, these primaries are important in determining if there may be reason why the ratings came out a certain way. However, Ostermeier never investigated that.

    "Your response reeks of tu quoque, and that's reading it charitably."
    Once again, my point was that if you use his methods to show possible bias in 2010, you have to also conclude no bias likely existed in 2007.

    "Of about 40 "False" and "Pants on Fire" statement rated in 2007 under half were penned by current PF staff (Adair, Drobnic Holan, Hollyfield)."
    So if you filter out previous Politifact staff, do the results look any different?

    "The editorial boards may be at least as important, but PolitiFact does not identify those (part of their program of transparency, doubtless)"
    So we are left with no conclusions either way. Helpful....

    "It's nice of you to volunteer help in undercutting your attempted comparison."
    I don't see how. This actually is an important point in my comparison. Could it be that the highly televised, highly emotional, and often furious TEA party contributed to the bad ratings?

    "Have I mentioned that I'm busy doing a research project for which PolitiFact is the subject? Trust me or don't."
    I have to say I don't. You strike me as someone who is more ideological than skeptical. Given your absurd responses (you have been a bit better in the conversations on my blog, so thank you for that), and lack of desire to use critical thinking skills when the result could possibly undermine your "Politifact is biased and basically incompetent" ideas, I see no reason to trust your research.

    "You can do your own research on it. Either 2008 or 2009 should serve. Or go for 2011."
    Okaythen. 2009 is likely to be similar to 2010. Dunno about 2008 or 2011 (although the republican presidential debates will probably make it look worse than 2011)

    ReplyDelete
  12. "The data he provides is for 2010, so at the very least he assumes bias in 2010. However, he also, as I have pointed out, assumes bias throughout Politifact's history."

    It's hard to see how your title is supposed to communicate the idea that Ostermeier assumes bias throughout PolitiFact's history. It's hard to see how Ostermeier's post communicates that idea, ftm. Is there some sort of secret code embedded in the words you used? Or did I not make it clear to you what I was asking?

    "I used the same methods as him."

    Only to a point. You employed your data stupidly while he did not. Plus I'm still wondering which independent received a "PoF" in 2007. If you had used your data for the same sorts of conclusions Ostermeier did then you'd have been fine. Instead you assume he draws conclusions about the whole of PF's history and you tried to use your data to show that he did something wrong. He didn't, unless we count acts performed only in your imagination.

    "But how do you know a significant number of the bad ratings for republicans didn't come from that?"

    Why would it matter? You seem to misunderstand the importance of party-on-party claims. They will tend to offer a more neutral palette. A person with a liberal bias may have no dog in the hunt at all for party-on-party claims. We'd expect them to exhibit less bias. It's all in who receives the damage. A "True" for a Republican in a party-on-party rating can damage Republicans generally quite well. See "Ron Paul." Paul is the truthful Republican--at least when he's not advocating Republican ideas. :-)

    "(M)y point was that if you use his methods to show possible bias in 2010, you have to also conclude no bias likely existed in 2007."

    Meh. You appear to have shifted your goalposts. Didn't you once claim that Ostermeier asserts bias throughout PF's history? It does follow that the type of bias Ostermeier tried to measure wasn't in evidence in 2007. But you've already had it explained to you why that would be the case. It was 100 percent primary season.

    "Could it be that the highly televised, highly emotional, and often furious TEA party contributed to the bad ratings?"

    What, by spiking journalistic bias against the tea party? ;-)

    Let's suppose it did what you suggest. That leaves us with one more factor the weakens your attempted comparison between 2007 and 2010. And it doesn't do a thing to undermine Ostermeier's thesis.

    "I see no reason to trust your research."

    *yawn*. I'm not asking you to trust my research. That's what "Trust me or don't" means.

    ReplyDelete
  13. "It's hard to see how your title is supposed to communicate the idea that Ostermeier assumes bias throughout PolitiFact's history."
    It doesn't just limit it to 2010. The reference to from the title is to the fact that the data from Ostermeier is from 2010. How much more do I have to dumb this down?

    "It's hard to see how Ostermeier's post communicates that idea, ftm. Is there some sort of secret code embedded in the words you used? Or did I not make it clear to you what I was asking?...
    Instead you assume he draws conclusions about the whole of PF's history and you tried to use your data to show that he did something wrong. He didn't, unless we count acts performed only in your imagination."

    I have already posted my response to this. Either respond to them or drop it.

    "Only to a point. You employed your data stupidly while he did not."
    Nope, mine was a near copy of his. Concepts are identical. Try again

    "Plus I'm still wondering which independent received a "PoF" in 2007."
    I counted Mike Gravel as an Independent since he was listed as a libertarian. I forgot he ran as a democrat in 2008. well that's 5/7 PoFs for Democrats that year.

    "Why would it matter? You seem to misunderstand the importance of party-on-party claims. They will tend to offer a more neutral palette. A person with a liberal bias may have no dog in the hunt at all for party-on-party claims. We'd expect them to exhibit less bias. It's all in who receives the damage. A "True" for a Republican in a party-on-party rating can damage Republicans generally quite well."
    By that same token, if a significant number of the more negative ratings from 2010 came from those primaries, it may be negatives that actually make republicans look good. So why would this not matter in 2010? In fact, if this is the case, then the negative ratings could have actually made republicans look good.

    "See "Ron Paul." Paul is the truthful Republican--at least when he's not advocating Republican ideas. :-)
    When he's wrong he's wrong. And he's actually about equal on his conservative/liberal claims. By I digress.

    "Meh. You appear to have shifted your goalposts. Didn't you once claim that Ostermeier asserts bias throughout PF's history?"
    Sorry that was sloppily worded:
    "(M)y point was that if you use his methods from the 2010 data, and you jump to conclusions about Politifact as a whole, you have to also jump to opposite conclusions using the data in 2007."
    (or you have to come up with some whole new theory about Politifact not being biased in 2007 but biased in 2010)

    "It was 100 percent primary season."
    Still sound like an excuse. Why did you not ask to ignore the primary statements from 2010? Why did you not look at other possible indicators of other factors at play in the ratings?

    "What, by spiking journalistic bias against the tea party?"
    Or maybe the TEA party just played fast and loose with the truth?


    "Let's suppose it did what you suggest. That leaves us with one more factor the weakens your attempted comparison between 2007 and 2010. And it doesn't do a thing to undermine Ostermeier's thesis."
    I already pointed out in my comprehensive analysis how it challenges his thesis. How much of my time are you going to waste repeating claims I have already addressed?

    ReplyDelete
  14. "*yawn*. I'm not asking you to trust my research. That's what "Trust me or don't" means."
    Not quite sure why you wrote this... You said "trust me or dont" and I said i "dont"

    ReplyDelete
  15. "It doesn't just limit it to 2010. The reference to from the title is to the fact that the data from Ostermeier is from 2010."

    The title doesn't say anything about data at all. It just says "Supposed Politifact Bias In 2010" respecting Ostermeier's work. You do remember criticizing Ostermeier's chosen title, don't you?

    "Either respond to them or drop it."

    I've responded. Still waiting for your response.

    "Nope, mine was a near copy of his."

    Rubbish. You applied your data as a comparison to his. His data stood alone and he compared it to no other year. There's no comparison in the two applications. I explained why, and you're not bothering to answer.

    "Mike Gravel"

    k, thnx.

    So why would this not matter in 2010?

    Nobody's saying it doesn't matter. It just isn't going to have a great effect because most of the 2010 claims are not primary claims (and a minority of those claims are party-on-party). You expect a bit of noise in the data, even in physical science.

    "you have to come up with some whole new theory about Politifact not being biased in 2007 but biased in 2010"

    To be clear, if we take your research as accurate, it shows perhaps a mild selection bias against Democrats in 2007, with the caveat that the sample size is small and primary claims serve as a more ambiguous measure of ideological bias than do general election claims and political speeches generally. The data in 2007 do not seriously affect reasonable conclusions drawn from the 2010 data.

    "Still sound like an excuse."

    An excuse for what?

    "Why did you not ask to ignore the primary statements from 2010?"

    For the same reason I didn't ask for the data in 2007 to be ignored. It's real data. It just isn't as useful in measuring ideological bias. The problem is the attempt to compare 2007 to 2010 without taking the differences into account. Ostermeier didn't do that.

    "maybe the TEA party just played fast and loose with the truth?"

    And PolitiFact's editors were able to reliably detect it in order to select it? If they can't, then I don't see the relevance.

    "I already pointed out in my comprehensive analysis how it challenges his thesis. How much of my time are you going to waste repeating claims I have already addressed?"

    What, I have to read everything on this blog before I can criticize it? ;-)

    I only gave your so-called "comprehensive analysis" a brief skim. It's unlikely to help your case upon closer examination.

    ReplyDelete
  16. Not quite sure why you wrote this... You said "trust me or dont" and I said i "dont"

    You went on at length. A "don't" requires no written response at all. But I think I know why you did it.

    ReplyDelete
  17. "The title doesn't say anything about data at all. It just says "Supposed Politifact Bias In 2010" respecting Ostermeier's work. You do remember criticizing Ostermeier's chosen title, don't you?"
    Again, I was using more than Ostermeier's title to show he was making generalizations about Politifact. You keep on reading that my title excludes all other years. Yet there is no indication of this. It does highlight that Ostermeier supposedly found evidence of bias in 2010, but It doesn't exclude the possibility he still made generalizations about Polifact as a whole . I'm not sure how much more I can dumb this down.

    "I've responded. Still waiting for your response."
    Where did you do this? I have responded to practically every word you have written in these comments.

    "Rubbish. You applied your data as a comparison to his. His data stood alone and he compared it to no other year. There's no comparison in the two applications. I explained why, and you're not bothering to answer."
    Yes I added the comparison, but I stated the data in a near identical fashion to his. Once again, what have I not responded to?

    "Nobody's saying it doesn't matter. It just isn't going to have a great effect because most of the 2010 claims are not primary claims (and a minority of those claims are party-on-party). You expect a bit of noise in the data, even in physical science."
    My point was that you did not filter out this noise. When the signal-to-noise ratio is too low, the data is not reliable.
    And it doesn't matter if they aren't MOST. For the purposes of argument, lets say there were 60 statements out of the over 300 total statements, ~20%. If these are all false and Pants on fire (remember R on R for these can be good, as you have already pointed out), that would cut the total down to 54 for Republicans vs 22 For Democrats. So the 3-to-1 Ratio now becomes 2-to-1. And this is only one factor out of many that could be controlled.

    "To be clear, if we take your research as accurate, it shows perhaps a mild selection bias against Democrats in 2007, with the caveat that the sample size is small "
    Actually the research could also have shown that Democrats were more likely to make false claims than republicans. And the sample size is not small. I already pointed out why earlier. Statistics 101, my friend.
    And once again, you would have to come up with a reason for why Politifact is biased against democrats in 2007, but biased against republicans in 2010.

    ReplyDelete
  18. "An excuse for what?"
    An ad-hoc excuse to ignore the data in 2007 for reasons you didn't apply to 2007 (investigating the nature of the statements).

    "For the same reason I didn't ask for the data in 2007 to be ignored. It's real data. It just isn't as useful in measuring ideological bias. "
    Basically, for the purposes of determining selection bias, you did ignore the data in 2007.

    "and primary claims serve as a more ambiguous measure of ideological bias than do general election claims and political speeches generally. The data in 2007 do not seriously affect reasonable conclusions drawn from the 2010 data."
    Once again, you are making an assumption about the nature of the claims in 2007. The point of my article is that there were plenty of other similar questions that had to be asked about the nature of the 2010 data as well. It is this kind of problem you say nullifies the 2007 data. But this kind of problem nullifies the 2010 as well.


    "The problem is the attempt to compare 2007 to 2010 without taking the differences into account. Ostermeier didn't do that."
    In other words, yo need to investigate the nature of the claims, which Ostermeier pretty much did not do in 2010.

    "And PolitiFact's editors were able to reliably detect it in order to select it? If they can't, then I don't see the relevance."
    It seems they were, see my last post as to possible teasons why this may be so.

    "What, I have to read everything on this blog before I can criticize it?"
    I never made any comment about what I think is Ostermeier's thesis in this article. If you want to say these things don't challenge Ostermeier's thesis, then you need to see where I actually use the data to challenge what I claim is Ostermeier's thesis.

    "I only gave your so-called "comprehensive analysis" a brief skim."
    Well that explains it.

    "It's unlikely to help your case upon closer examination."
    Sounds like you had no idea what I said if you actually think that.

    "You went on at length. A "don't" requires no written response at all."
    Unless I want to point out why I don't, but this is going off on a tangent.

    "But I think I know why you did it."
    Good for you :)

    ReplyDelete
  19. "You keep on reading that my title excludes all other years. Yet there is no indication of this. It does highlight that Ostermeier supposedly found evidence of bias in 2010 ..."

    I think it's fair to say that you can cram more into four words and a number than anyone in the history of the universe. Congratulations. One wonders why you bothered with the rest of the post when we can get all that from the title.

    "Where did you do this?"

    Hard to say at this point. Needle. Haystack. You find where you think you responded and I'll either find my response to it or craft a fresh one.

    "(W)hat have I not responded to?"

    You haven't responded to the fact that problems with your attempted comparison are not problems with Ostermeier's study, which attempts no such comparison.

    "My point was that you did not filter out this noise. When the signal-to-noise ratio is too low, the data is not reliable."

    So do you have evidence that the signal to noise ratio is too low?

    "And it doesn't matter if they aren't MOST"

    Your argument subsequent to this isn't clear, and it doesn't seem to make any sense at all when you're also apparently denying that primary claims are less definitive in measuring bias than other claims.

    "And the sample size is not small.

    Point being that the margin for error may encompass the estimated disparity. The sample size encompasses all of the 2007 data, which is good, but that's not exactly where we measure for the disparities. That occurs in smaller subgroups (such as "Pants on Fire" ratings for which only a few from each party were represented).

    "An ad-hoc excuse to ignore the data in 2007 for reasons you didn't apply to 2007"

    But I don't ignore the 2007 data. I just discourage comparing 2007 to 2010 as though it's apples-to-apples. I called it a statistical outlier simply because it's so different from every subsequent year.

    "Basically, for the purposes of determining selection bias, you did ignore the data in 2007."

    Where?

    "Once again, you are making an assumption about the nature of the claims in 2007."

    Dang it! I thought I remembered reading tons of PF stories and thinking things through and now to find out none of it ever happened. Thanks so much for opening my eyes.

    "In other words"

    Wow. What you did with your own title you can also do with the words of others.

    "I never made any comment about ... "

    Right. How about extending the courtesy you expect to others? Ostermeier never made any comment about applying the conclusions of his 2010 data to other years.

    "Sounds like you had no idea what I said if you actually think that."

    You have a record. It's not good.

    ReplyDelete
  20. "I think it's fair to say that you can cram more into four words and a number than anyone in the history of the universe. Congratulations. One wonders why you bothered with the rest of the post when we can get all that from the title."
    You are the one saying it excludes all years but 2010. I am saying you cannot come to that conclusion and why you cant.

    "Hard to say at this point. Needle. Haystack. You find where you think you responded and I'll either find my response to it or craft a fresh one."
    So you want me to do your work for you? Great job in shifting the burden. I have done a point by point reply to every comment you have made so I'm skeptical of any such ignored argument.

    "You haven't responded to the fact that problems with your attempted comparison are not problems with Ostermeier's study, which attempts no such comparison."
    Dear Gawd, once again, I showed those comparisons to point out problems with Ostermeier's generalizations. I have said this numerous times, have you forgot? I feel like a broken record. Ostermeier used the raw numbers without investigating the nature of the statements, so I did the same to show internal contradictions in his generalizations. You seem to have at least partially figured this out since you are asking about the nature of the statements in 2007.

    "So do you have evidence that the signal to noise ratio is too low?"
    I didn't say it was. I just said what would happen if it was. I don't know what the signal to noise ratio is in this case. I'm just saying you need to know before you ignore the noise.

    "Your argument subsequent to this isn't clear, and it doesn't seem to make any sense at all when you're also apparently denying that primary claims are less definitive in measuring bias than other claims."
    I thought this was painfully clear, but ill summarize again:

    If primary claims are less definitive AND a significant number (not necessarily a majority) of false-er claims in 2010 were in fact primary claims, then you cannot ignore their effects on the distribution. You have to filter them out, just like you are with 2007.

    "Point being that the margin for error may encompass the estimated disparity."
    It may as well with 2010, although it is a bit less likely, it is not all that much.

    "The sample size encompasses all of the 2007 data, which is good, but that's not exactly where we measure for the disparities. That occurs in smaller subgroups (such as "Pants on Fire" ratings for which only a few from each party were represented)."
    So were only looking at one subgroup? Other than his last statement, Ostermeier measures in much larger subgroups, such as "mostly false" and below.

    "But I don't ignore the 2007 data. I just discourage comparing 2007 to 2010 as though it's apples-to-apples. I called it a statistical outlier simply because it's so different from every subsequent year."
    That is essentially ignoring it. It is only there for comparison. It is apples-to-apples because Ostermeier never looks at the nature of the claims, filtering out the kinds of claims that one would expect to create noise if one were trying to determine is party-related selection bias exists.

    ReplyDelete
  21. "Where?"
    look up.

    "Dang it! I thought I remembered reading tons of PF stories and thinking things through and now to find out none of it ever happened. Thanks so much for opening my eyes."
    Great quote mining job, here's the rest of the statement:
    "The point of my article is that there were plenty of other similar questions that had to be asked about the nature of the 2010 data as well. It is this kind of problem you say nullifies the 2007 data. But this kind of problem nullifies the 2010 as well."

    "Wow. What you did with your own title you can also do with the words of others."

    You said: "The problem is the attempt to compare 2007 to 2010 without taking the differences into account. Ostermeier didn't do that."
    Does the difference not have to do with the nature of the claims? It sure seems like pointing out that the claims were mostly D-on-D and R-on-R is a perfect example of investigating the nature of the claims. So back to my point you just dodged for some reason:
    "In other words, yo need to investigate the nature of the claims, which Ostermeier pretty much did not do in 2010."
    You investigated the nature of the claims. The thing I keep pointing out, but yet you love to dodge, is that neither you nor Ostermeier investigated the nature of the claims.

    "Right. How about extending the courtesy you expect to others? Ostermeier never made any comment about applying the conclusions of his 2010 data to other years."
    I already pointed out how he both inferred it later on in the article and how he explicitly endorsed it with his last paragraph. Where is there evidence of me inferring anything about Ostermeier? All I did was point out how anti-fact-checkers such as Hemingway were using this article to claim party-related selection bias on the part of Politifact.

    "You have a record. It's not good."
    I have a record of not responding well to absurdities and contrived arguments.

    ReplyDelete
  22. "You are the one saying it excludes all years but 2010."

    Well, no, I'm not saying that. I'm just saying that your words look specific to 2010. You're the one saying the title only refers to the data Ostermeier collected in 2010. I'm simply pointing out that it's hard for those who lack your superhuman skill set to detect that in the title. It seems to me that the title "Supposed PolitiFact Bias nonexistent in 2007" communicates the idea you're claiming for it much better to us mere mortals. Once you say "2010" it makes it seem as though you're talking about 2010.

    "So you want me to do your work for you?"

    Well, no. I'm not asking you to find my response. I'm asking you to find the one you said you'd written. You'd only be supporting your own claim. Once you do that I'll either support mine or supply the response you're hoping to engage. More rocket science?

    "I showed those comparisons to point out problems with Ostermeier's generalizations."

    You're hilarious.

    You chose a sample that consisted entirely of primary season claims. Is that typical of PolitiFact? Or is a mixture of claims more typical? Sheesh.

    Allow me to sum it up for you. Ostermeier collected data for slightly over a year and published soon after. Even an idiot knows that he's talking about the picture of PolitiFact at the time he's writing based on his data from the preceding year. Recent data are the best data to draw from in assessing PF as of the time he wrote. Your supposition that he was talking about all of PF's history is absolutely the worst type of straw man attack. If Ostermeier had written in early 2011 relying on data from any one full year aside from the one he used then he'd be rightly criticized.

    Fixed this for you:

    "I have a record of responding with absurdities and contrived arguments."

    The logical proof you left at my blog is a perfect example. How does one use sets to describe net job creation? What is the set of negative jobs created? It's absurd to use a set for that type of argument, yet that's exactly what you did. There's no absurdity involved in Texas having (for example) a net creation of 10 million jobs and the U.S. overall having a net creation of 5 million (or even a net loss nationally). And we can express it coherently in terms of percentages. Will that percentage comparison confuse many people because it's not the way we usually refer to negatives? Sure. But that doesn't make it incoherent or absurd. The math works fine. Your logic didn't because you treated net job creation as though it was gross job creation.

    You know what that argument says about you? And you're still at it, with more of the same in the oven and on the cooling racks. You're like Ed Wood, telling yourself that you're doing wonderful work. It's marvelous, really.

    ReplyDelete
  23. This comment has been removed by the author.

    ReplyDelete
  24. This comment has been removed by the author.

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. ”Well, no, I'm not saying that. I'm just saying that your words look specific to 2010. You're the one saying the title only refers to the data Ostermeier collected in 2010.”

    You compared my title to Ostermeiers, which makes a generalization about Politifact (which is repeated throughout much of the article). Mine pointed out a fact about 2010, but did not exclude any other years. The comparison only works if it honestly appears as though my title excludes other years.

    ”I'm simply pointing out that it's hard for those who lack your superhuman skill set to detect that in the title. It seems to me that the title "Supposed PolitiFact Bias nonexistent in 2007" communicates the idea you're claiming for it much better to us mere mortals. Once you say "2010" it makes it seem as though you're talking about 2010.”

    Not if you read Ostermeier’s article, it should be pretty clear what I’m talking about.

    ”Well, no. I'm not asking you to find my response. I'm asking you to find the one you said you'd written. You'd only be supporting your own claim. Once you do that I'll either support mine or supply the response you're hoping to engage. More rocket science?”

    Wow, you really are a rhetorical trickster aren’t you. I said I assumed I probably did respond to whatever point you said I didn’t respond to. I do not know. I just said I doubt it because I’ve done point by point critiques of your arguments. Once again, you made the claim that you had some argument I never responded to. Why do you keep weaseling out of providing evidence to back up your claims?

    ReplyDelete
  27. ”Well, no. I'm not asking you to find my response. I'm asking you to find the one you said you'd written. You'd only be supporting your own claim. Once you do that I'll either support mine or supply the response you're hoping to engage. More rocket science?”

    Dear gawd I really do feel like a broken record. I did EXACTLY WHAT OSTERMEIER DID. I chose a year, IGNORED THE NATURE OF THE CLAIMS, JUST LIKE OSTERMEIER DID, and presented the data in a near mirror image of the way Ostermeier did. My comparisons were there to show the problems of essentially cherry picking data and, wait for it….. IGNORING THE NATURE OF THE CLAIMS.

    ”Allow me to sum it up for you. Ostermeier collected data for slightly over a year and published soon after. Even an idiot knows that he's talking about the picture of PolitiFact at the time he's writing based on his data from the preceding year. Recent data are the best data to draw from in assessing PF as of the time he wrote. Your supposition that he was talking about all of PF's history is absolutely the worst type of straw man attack. If Ostermeier had written in early 2011 relying on data from any one full year aside from the one he used then he'd be rightly criticized.”

    Not if the nature of the claims may be the cause of the data being that way. Let me give you an analogy. Let’s say I’m looking at a race horse that has been around for 4 years at the same race track. I look at the data from the last year and see that the race horse has been winning a lot of races. However, I see that about two years ago, all the best racehorses went to a different race track and were replaced with incredibly inferior race horses. However, despite this crucial piece of information, I conclude this particular race horse must be really good. As you can see, merely looking at JUST the most recent data is not good enough to come to a conclusion about the race horse itself. Now, look at how Ostermeier notices one key piece of information:

    What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans.

    Yet he ignores how this might affect the data. This was a major point of my other article. Instead he makes some snide comment about politifact being biased against the party out of power, which turns the focus onto politifact instead of investigating the nature of the claims. Looking at the data in 2007 at least partially controlled for this factor. If we had a year in politifact’s history where Republicans had full control of government, I would have looked at that year instead. However, 2007 was the closest I could find.

    ReplyDelete
  28. ”The logical proof you left at my blog is a perfect example. How does one use sets to describe net job creation? What is the set of negative jobs created? It's absurd to use a set for that type of argument, yet that's exactly what you did.”

    My logical proof had to do with how people interpret the statement
    "Approximately x percent of the a were in b (with b existing in a)."

    “approximately 20 percent of the apples put into the bag were red apples.”

    “approximately 10 percent of the houses built in the neighborhood were built by me.”

    People think of statements like these in terms of collections. Seeing as how these collections can be reasobaly expected to not hold two of the exact same element, sets work just fine. As you seem to understand, negatives cannot be represented as the length of a collection, creating a problem in how people would naturally interpret a statement.

    “approximately 50000 percent of the apples put into the bag were red apples.”

    “approximately 10000 percent of the houses built in the neighborhood were built by me.”

    Yet you seem to not understand how absurd a statement like these sound:

    "Approximately 50000 percent of the jobs created in the U.S. from November 2007-2008 were in Texas."

    "Approximately -48000 percent of the jobs created in the U.S. from November 2007-2008 were in Texas."

    Both sound, at the very least misleading, at worse absurd. I had to use that proof because you for some reason didn’t seem to get just how absurd these statements sound, which is itself absurd.

    ”There's no absurdity involved in Texas having (for example) a net creation of 10 million jobs and the U.S. overall having a net creation of 5 million (or even a net loss nationally).”

    Yes but, as I pointed out, Perry didn’t phrase it that way. There are some ways to present that data that are misleading at best, unnatural and absurd-sounding at worst. He chose one of those ways.
    ”And we can express it coherently in terms of percentages. Will that percentage comparison confuse many people because it's not the way we usually refer to negatives? Sure.”

    When you include negative job creation, yes percentages are a terrible way to express it. It can make the statement ambiguous.

    ”But that doesn't make it incoherent or absurd. The math works fine. Your logic didn't because you treated net job creation as though it was gross job creation.”

    I treated it in a way people would be expected to understand it. I’ve pointed this out numerous times in the other post, yet you seem to ignore me. My logic only doesn’t work when you completely bungle your interpretation of what I said.

    ”You know what that argument says about you? And you're still at it, with more of the same in the oven and on the cooling racks. You're like Ed Wood, telling yourself that you're doing wonderful work. It's marvelous, really.”

    No matter how much I explain these unbelievably simple concepts to you, you still seem to have an endless supply of absurdities, rhetorical tricks, and selective interpretations to avoid what even the most modest critical thinker should be able to see. This is why I gave up on your old post. You seriously are a real piece of work. That was cute what you did with “fixing” my statement for me. I almost forgot most of my memories from the fifth grade. Thanks for jogging them.

    ReplyDelete
  29. I thought I might nip this point in the butt:

    And we can express it coherently in terms of percentages. Will that percentage comparison confuse many people because it's not the way we usually refer to negatives? Sure.

    As I pointed out, the reason it confuses people is because it's not the way people would normally translate the kind of statement Perry makes. When they translate it that way, they end up with an absurdity. Now people may be inclined, due to the absurdity to find another way of translating the statement in a way they wouldn't normally. That may lead them to a translation of the statement that is actually not absurd, but makes perfect sense. However, the point I wanted to make with this hypothetical is that they normally wouldn't see the need to translate the statement outside they way they normally do if there was no absurdities:

    Now, based on Perry's claim "Approximately 70 percent of the jobs created in the U.S. from November 2007-2008 were in Texas.," people would naturally think he created a majority of the jobs in the US during that period, since we normally think of a majority as over 50%. When someone does a majority of something, we obviously think no other single person did more than them. Now lets take another hypothetical:

    Texas created 70,000 jobs during Time Period a. Colorado created 80,000 jobs during Time Period a. All other 48 states states combine for a net loss of 50000 jobs, meaning the net job creation of the entire US during Time Period a is 100,000. The governor of Texas could also say "Approximately 70 percent of the jobs created in the U.S. during Time Period a were in Texas." Of course, this would be misleading in the normal way people translate the statement since Colorado actually created more jobs than Texas, so obviously Texas could not have created the majority of jobs in the US, as the statement suggests. However, since this statement doesn't look absurd (without context information about Colorado), the reader would have no reason to translate the statement beyond their normal interpretation.

    So as you should be able to plainly see, the kind of way Perry presented the data is problematic when used to represent net jobs. (of course, this is just one of many reasons Perry get a "false" from Politifact)
    http://www.politifact.com/texas/statements/2010/jan/12/rick-perry/perry-claims-texas-accounted-70-percent-new-jobs-2/

    ReplyDelete
  30. "You compared my title to Ostermeiers, which makes a generalization about Politifact (which is repeated throughout much of the article)."

    Again, you're simply hilarious. Ostermeier's title:

    Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.

    What's the generalization?

    ReplyDelete
  31. "Again, you're simply hilarious. Ostermeier's title:

    Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats.

    What's the generalization?"

    Are you kidding? the whole thing is a generalization.

    ReplyDelete
  32. No, I'm not kidding. If the answer is easy then it's all the more reason for you to give it.

    Is "Selection Bias?" a generalization? If yes, what is the content of the generalization in your opinion?

    ReplyDelete
  33. "Selection Bias?" alone is not a generalization. However, the whole statement together is.

    If Politifact has been doing this continuously, then there's no indication he's talking about just 2010. If fact checking operations had been done in a discrete manner, such as specific seasons (or games for a sports team), then it is possible someone wouldn't necessarily see it as a generality. Looking at the last year is an arbitrary time period.

    ReplyDelete
  34. I read back over the article, and realized this whole argument over whether or not Ostermeier was generalizing is mostly trivial:

    "So his conclusions are good for 2010, obviously. 2007 data do not matter with respect to 2010."

    As I pointed out in my findings:

    "As you can see, the results from 2007 can seriously undermine many possible conclusions that a person could draw from the 2010 data."

    This includes the issue of party-related selection bias. Ostermeier did in fact note something special about 2010:

    "What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans."

    Yet he does not even suggest that this may be a cause of the disparity, outside suggesting some odd kind of selection bias. As a result, the best kind of control he could have offered would have been to examine another year where Democrats did not have full control. The years 2003-2005 (ignored 2006 due to various reasons), would have been perfect since the situation was in fact the mirror opposite. However, he could have settled by checking another year when neither party was truly in charge (2007 or 2008). Seeing that by his same standards no disparity existed in 2007, he would have to come up with some contrived theory over Politifact becoming biased during that time period just to ensure his bias theory still has any credence (or look at the nature of the claims, which would have challenged any possible conclusion of selection bias as well). I'd venture to guess such an ad-hoc explanation would sound a bit fishy to say the least.

    The reason I pointed out that his methods used selection bias was because 2010 was no arbitrary year. It is the most recent of a continuous operation, which means the start and end dates he chose can possibly drastically affect the results.

    ReplyDelete
  35. "'Selection Bias?' alone is not a generalization. However, the whole statement together is."

    Then what is the supposed generalization for the whole statement?

    ReplyDelete
  36. "Then what is the supposed generalization for the whole statement?"
    Already answered this question a few times, for instance:

    "t depends on the population he is looking at. He actually is not completely consistent over his population. Notice the title says nothing about 2010. The suggestions he investigates say nothing about 2010 in particular. They are generalizations about Politifact's selection process. In fact, the interview he gets from Bill Adair comes from 2009. If he was worried about JUST 2010, why would he bring up an interview from 2009? He clearly, throughout the entire article, talked about possible selection bias in the way Politifact selects which statements to read. This applies to all years Politifact has been around, including 2007."

    Of course, as I pointed out, the whole point is trivial as my objections would still be valid even if he was only talking about 2010 (see previous comment).

    ReplyDelete