I checked Twitter this morning and found this from Brendan Nyhan, aka "my hero":

It's a post by John Sides at The Monkey Cage about a recent press release by the Center for Media and Public Affairs (CMPA) on the fact checking reports of PolitiFact. Basically, CMPA researcher cross-tabulated the percentage of statements ranked by PolitiFact fact checkers as true or false with the political party of the person who made the statement. Republicans' statements were rated as false with a 3 to 1 margin compared to Democrats, and Democrats' statements were rated as true with a 2 to 1 margin compared to Republicans.

CMPA hinted (but, as usual, did not outright say) that these results arise (at least in part) from PolitiFact's liberal bias. U.S. News & World Report contributing editor Peter Roff took that idea and ran with it, writing:

But, as Sides wrote at The Monkey Cage:

I've said as much when I calculated the possible extent of both liberal and centrist bias in statements by both PolitiFact and Glen Kessler, The Fact Checker at WaPo (who is very sad that Michelle Bachmann isn't running again). I also debated with some conservatives about how uncertain we are in the extent and type of bias among fact checkers.

But while I agree that we don't know how biased fact checkers are (simply because we've never measured their bias relative to some reasonable baseline), I've always disagreed with John Sides' and Brendan Nyhan's opinions that the newsworthiness bias among fact checkers makes the sample of fact checked statements too biased for comparisons to be useful.

Fact checkers cover statements that actually matter to people because they want people to read what they write. What matters to the public is the truthfulness of the statements that politicians make about issues that we find important. If we just took a random sample of political statements, we'd come across a lot of innocuous ones about things that aren't that important. While a systematic analysis could come up with some reasonable way to weight statements by the importance of the issue to the public (or by some other rubric), looking at fact checkers' statements is a good first pass.

As I design the methods for SoundCheks, a fact checking (well, fallacy checking) research institute I dream of founding and making successful, I'll think a lot about my statement sampling methods. And I've always been thinking about how to measure fact checker bias compared to nonprofessionals and obvious partisans.

Finally, based on his interpretation of the evidence, Mooney advocates that we ignore the comments section.

I agree with Mooney that flame wars are detrimental to rational discourse, and that the George Mason study highlights the pitfalls of what Daniel Kahneman calls "System 1" thinking. Yet I counter that a purely civil discussion about climate change may also be counter-productive. Furthermore, the comments section doesn't mark a huge departure from "your father's media environment". Finally, the George Mason University study demonstrates that there is good reason to pay very close attention to the comments section, even if it is littered with bridges beneath which trolls dwell.

How can civil discourse be counter-productive? Agreeable people tend to groupthink, which is when people make poor decisions for the sake of harmony and conformity. Another possible consequence of over-civility is the middle ground fallacy, which is the false assumption that the middle point between two extremes must be the truth (medio tutissimus ibis, anyone?). A flame war presses emotional buttons that polarize a discussion and inhibit rationality. But an overly civil discourse massages the human tendency to conform, which may also lead us astray. The key is to balance investment in our beliefs with the willingness to abandon invalid arguments and discard false premises.

As for whether or not it is still your father's media environment, of course it isn't. Still, flame wars are anything but new. Trust me. I just spent eleven months in a rural village where people have limited access to electronic media, much less the Internet. People don't have to hide behind anonymous screen names to behave immaturely during a heated debate, quickly drowning out the radio program that prompted the discussion.

Finally, why shouldn't we ignore the comments section? It's not because we're likely to find high quality debate there. We should steel ourselves against the polarizing effects of flame wars, becoming part of the mob to better understand how people form and defend their beliefs. We shouldn't limit ourselves to observing the effects of flame wars from afar, as did the George Mason research team. There's something to be said for allowing yourself to get bated by trolls a few times to experience for yourself how easy it is to be led astray. If you look at my user history at Reddit or the Daily Kos, you'll see I speak from experience. I've learned valuable lessons about the limits of my rationality on those sites. We anthropologists would call this method of inquiry participant observation.

Apart from these three criticisms, I read Mooney's article and the research it covers in light of one of Malarkometer's missions, which is to quantify and correct for the bias and uncertainty inherent in measures of political figures' factuality. Another of my goals is to eventually host a site where both professional fact checkers and non-professionals engage in fact checking and discourse about fact checking while regularly answering questionnaires about political philosophy. The idea is to compare the influence of political philosophy on fact checking performance of professionals versus non-professionals. Based on the George Mason group's findings, I might also want to experiment with varying the levels of comment moderation across the site, and then examine the influence of forum comment vituperativeness on aggregate fact checking performance. And, yes, I might also want to suggest to my future fact checking staff to kindly avoid the comment section while they write their reports!

Republicans and maybe Independents with greater political knowledge perceive the unemployment statistics to be less accurate.

Here's a similar plot showing the percent in each political knowledge and party affiliation group that believe in the conspiracy theory about the September unemployment statistic report.

Democrats appear less likely to believe the conspiracy theory the more knowledgeable they are. Republicans with greater political knowledge are more likely to believe the conspiracy theory. There's no clear effect among Independents. What's going on?

Perhaps the more knowledgeable individuals are also more politically motivated, and so is their reasoning. It just so happens that motivated reasoning in this case probably errs on the side of the politically knowledgeable Democrats.

Before discussing what this means for fact checkers and factuality metrics, let's look at what voteview writes about an aggregate answer to a different question, posed by Gallup (aka, the new whipping boy of the poll aggregators) about the June jobs report.

In case you haven't figured it out, you're looking at yet another picture of motivated reasoning at work (or is it play?). Democrats were more likely than Republicans to see the jobs report as mixed or positive, whereas Republicans were more likely than Democrats to see it as negative. You might expect this effect to shrink among individuals who say they pay very close attention to news about the report because, you know, they're more knowledgeable and they really think about the issues and... NOPE!

The more people say they pay attention to the news, the more motivated their reasoning appears to be.

What's happening here? In Nyhan's study, are the more knowledgeable people trying to skew the results of the survey to make it seem like more people believe or don't believe in the conspiracy theory? In the Gallup poll, is "paid very close attention to news about the report" code for "watched a lot of MSNBC/Fox News"? Or is it an effect similar to what we see among educated people who tend to believe that vaccinations are (on net) bad for their children despite lots and lots of evidence to the contrary? That is, do knowledgeable people know enough to be dangerous(ly stupid)?

I honestly don't know what's happening, but I do have an idea about what this might mean for the measurement of potential act checker bias to aid the creation of better factuality metrics and fact checking methods. I think we can all agree that fact checkers are knowledgeable people. The question is, does their political knowledge and engagement have the same effect on their fact checking as its does on the perceptions educated non-fact-checkers? If so, is the effect as strong?

I've mentioned before that a step toward better fact checking is to measure the potential effect of political bias on both the perception of fact and the rulings of fact checkers. Basically, give individuals a questionnaire that assesses their political beliefs, and see how they proceed to judge the factuality of statements made by individuals of known party affiliations, ethnicity, et cetera. To see if fact checking improves upon the motivated reasoning of non-professionals, compare the strength of political biases on the fact checking of professionals versus non-professionals. 

What these two blog posts tell me is that, when drawing such comparisons, I should take into account not only the political affiliation of the non-professionals, not only the political knowledge of the non-professionals, but the interaction of those two variables. Then, we can check which subgroup of non-professionals the professional fact checkers are most similar to, allowing us to make inferences about whether professional fact checkers suffer from the same affliction of motivated reasoning that the supposedly knowledgeable non-professionals suffer from.

This is Internet commenting at its best: constructive, well-reasoned, and mainly correct. Let's address the comment point by point.

"Validity is a nice standard for mathematics and logic but it is not often found in public discourse."

I can't agree more. This unfortunate fact should not, however, discourage us from specifying and enumerating the logical fallacies that public figures commit. It should encourage us to do so, as it has encouraged the establishment of the fact checking industry.

"Even scientific conclusions are rarely (if ever) backed by valid reasoning as they typically rely on induction or inference to the best explanation."

I agree that scientists stray from valid (and sound) argumentation more often than they should. I do not, however, agree that scientists rarely if ever make sound or valid arguments. I also agree that scientists often use inductive reasoning. Scientists will continue to do so as Bayesian statistical methods proliferate. I do not, however, agree that inductive inference is immune to the assessment of soundness and, by inclusion, validity. Inductive reasoning is probabilistic. For instance, a statistical syllogism (following Wikipedia's example) could go,

1. 90% of humans are right-handed.
2. Joe is a human.
3. Therefore, the probability that Joe is right-handed is 90% (therefore, if we are required to guess [one way or the other] we will choose "right-handed" in the absence of any other evidence).

You can assess the validity of this statistical syllogism by considering whether the steps in the argument follow logically from one another. You can assess its soundness by furthermore considering whether its premises are true. Are 90% of humans right-handed? Is Joe a human? Inductive logic is still logic.

"Not every claim is an argument. An argument must offer evidence intended to support a conclusion. I can claim 'I am hungry' without thereby offering any sort of argument (valid, inductive, fallacious or otherwise) in support of that claim. One cannot test the validity of a single proposition."

I agree that not every claim is an argument, either in the formal or informal sense. Every claim is, however, a premise. In such cases, we can simply determine whether or not the premise is true. Furthermore, many claims that fact checkers care about imply or support an informal (or even formal or legal) argument. In such cases, you can assess the implied informal argument's validity. Lastly, in any case where a public figure makes a claim that ties vaguely to an informal argument, that public figure deserves to be criticized for committing the ambiguity fallacy. Many politicians often commit the ambiguity fallacy. As much as possible, we should call them on it whenever they do it.

"No need to check for 'both' soundness and validity. If you check for soundness, then you have already checked for validity as part of that. Perhaps you meant to say you would check for both truth of basic premises and validity of reasoning."

Correct. To be sound, an argument must be valid. What I should have said is that fact checkers conflate truth with validity.

"It depends a bit on which notion of fallacy you are working with, but arguments can fail to be valid without committing a common named fallacy. A far simpler check for validity is simply to find counterexamples to the reasoning (logically possible examples in which the basic premises of the argument are all true and in which the conclusion of the argument is false)."

I hope that Ima Pseudonym will elaborate on the logical counterexample part of this statement. If it's a viable shortcut, I'm all for it. That said, I suspect that there are many logical fallacies that do not yet have a name. Perhaps Malark-O-Meter's future army of logicians will name the unnamed!

Thank you again, Ima Pseudonym. Your move if you wish to continue playing. I like this game because you play it well. I encourage constructive criticism from you and all of Malark-O-Meter's readers. Cry 'Reason,' and let slip the dogs of logic.

Whether or not fact checkers wield it as a "moral weapon", they certainly use the "language of truth and falsehood", and some of them attempt to define "bright-line categories". This is most true for PolitiFact and The Fact Checker, which give clear cut, categorical rulings to the statements that they cover, and whose rulings currently form the basis of Malark-O-Meter's malarkey score, which rates the average factuality of individuals and groups. 

The language of truth and falsehood does "invoke ideas of journalists as almost scientific fact-finders." But it isn't just the language of truth and falsehood that bestows upon the art of fact checking an air of science. Journalists who specialize in fact checking do many things that scientists do (but not always). They usually cover falsifiable claims, flicking a wink into Karl Popper's posthumous cup of tiddlies. They always formulate questions and hypotheses about the factuality of the claims that they cover. They usually test their hypotheses against empirical evidence rather than unsubstantiated opinion.

Yet Fact checkers ignore a lot of the scientific method. For instance, they don't replicate (then again, neither do many scientists). Moreover, fact checkers like PolitiFact and The Fact Checker use rating scales that link only indirectly and quite incompletely to the logic of a claim. To illustrate, observe PolitiFact's description of its Truth-O-Meter scale.

Sometimes, fact checkers specify in the essay component of their coverage the logical fallacies that a claim perpetrates. Yet neither the Truth-O-Meter scale nor The Fact Checker's Pinocchio scale specify which logical fallacies were committed or how many. Instead, PolitiFact and The Fact Checker use a discrete, ordinal scale that combines accuracy in the sense of correctness with completeness in the sense of clarity. 

By obscuring the reasons why something is false, these ruling scales make it easy to derive factuality metrics like the malarkey score, but difficult to interpret what those metrics mean. More importantly, PolitiFact and The Fact Checker make themselves vulnerable to the criticism that their truth ratings are subject to ideological biases because...well...because they are. Their apparent vagueness makes them so. Does this make the Truth-O-Meter and Pinocchio scales worthless? Probably not. But we can do better. Here's how.

When evaluating an argument (all claims are arguments, even if they are political sound bites), determine if it is sound. To be sound, all of an argument's premises must be true, and the argument must be valid. To be true, a premise must adhere to the empirical evidence. To be valid, an argument must commit no logical fallacies. The problem is that the ruling scales of fact checkers conflate soundness and validity. The solution is to stop doing that.

When and if Malark-O-Meter grows into a fact checking entity, it will experiment with rating scales that specify and enumerate logical fallacies. It will assess both the soundness and the validity of an argument. I have an idea of how to implement this on the web that is so good, I don't want to give it away just yet.

There are thousands of years of formal logic research that stretch into the modern age. Hell, philosophy PhD Gary N. Curtis publishes an annotated and interactive taxonomic tree of logical fallacies on the web.

Stay tuned to Malark-O-Meter, where I'm staging a fact check revolution.