Somebody named Jared has been posting insightful comments on one of my earliest Malark-O-Meter posts on my other website. I decided to move the conversation here. Below, I address his most recent comment. He is in italics. I am in bold.
I know I know. I got so excited to find your blog, which again is awesome! Why so many in the media choose not to do this type of analysis confuses me.

One reason the media hasn't done a lot of analyses like this is because they are skeptical of what the average truthfulness measures. The thing is, so am I! But I am skeptical of almost everything that hasn't been systematically analyzed.

There is a good article by Michael Sherer on Time's Swampland blog in which he interviews the biggest names in fact checking about the difficulties with measuring individual truthfulness. My first mission with Malark-O-Meter is to eventually address each of the issues the fact checkers raise in that article. And by that, I mean refuting the claim that they make truthfulness scores meaningless. My second mission with M-O-M (hee hee, Mom!) is to actually show people, with analysis and graphs, how useful truthfulness scores are and are not, rather than just wave my hands in the air.

So I am glad you're on board! On with the rest of your comments.

1. What I realized right after my last post was, standard deviations from what? Normal truthfulness or political truthfulness? There might be already some data collected through previous research on truthfulness, my background is in pyschology and admittedly I took stats and psych stats as an undergrad and have since not retained as much as I would have liked. The other option would be to do an aggregate of statements from other american political figures or previous presidential candidates to get some normative political data for comparision. This might be more of an undertaking than you would be prepared for, and I am not sure if in fact there is data collected for that.



That is my plan! Eventually, I will write a web scraping script that collects all Pinocchio Tracker, PolitiFact, and other fact checker scores together. Then I can use that data set to create an observed probability distribution of truthfulness. From this observed probability distribution, I could give people percentile rankings. The cool thing is that percentile rankings are themselves a random variable! And I know how to calculate the uncertainty in percentiles, which I would then report. So, the data is there, and I am (eventually) going to use it. For now, we're going to look at absolute bullpucky scores and their comparisons.


2. Again I apologize for my ignorance here. I know from my background that a couple things that experimental psychologists have to deal with is rater error, and social desirability bias. I guess what I'm accusing Politifact of is riding the fence so to speak. I am pre-supposing a theoretical mean of .50 truthfulness and I'll guesstimate a standard deviation of .10 for arguments sake that would mean that at .54 Obama is just slightly better than average on accurate statements, and at .35 Romney would be a more than a standard d from normal honesty. Ok this is all nonsense because I'm making up my own numbers. Anyway, based on your data analysis one could reasonably assume that on a given number of talking points, Obama is more factual than Romney. That being said, the debate data shows both candidates being very close to the mean, my theoretical mean, but closer no matter how you view it. So my question then is, was Romney being more accurate than his statistical history or was there some sampling error in the data? There are ways to test a group of data points for sampling error using a chi squared, but again it's been so long. (my underline)


I haven't yet done the analysis using my new scoring and simulation methods, but I'll copy-paste my analysis results from my 1st presidential debate (and my first ever Malark-O-Meter-esque) statstical analysis here:


  1. We can be 95% confident that Barack Obama's credibility was between ~82% of what it normally is and over 2.5 times what it normally is. (So, yes, there is a lot of sampling error)
  2. The probability is over 80% that Barack Obama's overall credibility is greater than his 1st debate performance. (We can be somewhat confident he was less truthful than usual, but not certain.)
  3. We can be 95% confident that Mitt Romney's credibility was between ~52% and more than ~2 times what it normally is. (Again, lots of sampling error.)
  4. The probability that Mitt Romney's credibility was higher than normal is over 60%. (Note: this is different than what I wrote in the article because I accidentally flipped the inference around. LOL. Give me a break! It was the first analysis I did! Anyway, we can't be very certain of it, but maybe Mitt Romney was more truthful than usual.)


I don't do frequentist hypothesis testing (i.e., assign a p-value that says, yup, he's being more truthful!). But I do calculate the probabilities that we would get a certain truthfulness measure or truthfulness comparison measure from a sample of the size we have, given what we have observed (I don't yet make any explicit prior probability assumptions). Basically, I simulate the distribution and either integrate probabilities or calculate percentiles of the distribution.


So does this analysis answer your question?

3. I disagree! First of all the N is so much higher for overall statements. Of course, that means higher reliability statistically. What accounts for this variability in the debates? I agree more scrutiny. Perhaps, Romney chose his answers more carefully than when he is "firing up the crowd" at stump speechs and advertising. I feel it necessary at this point to say I'm not using your blog to bolster support for either candidate. I am just fascinated by the numbers here. 


You know what. You're right. Even though we have a higher sampling fraction for debate statements, it provides no advantage because we still only have a sample of rated statements. It also provides no disadvantage. Since we are sampling with replacement, we don't need to do any finite population correction. The thing that matters, as you point out, is sample size.

As to you last statement. Anything I can do to encourage you to keep it up! It has worked! I don't know why I was suprised by this but the highest percent of both candidates statements was half truths, around 25%. LOL, how political of them. My assumption was more polarizing. I assumed that both were mostly telling the truth or full of it than the data shows. 

Yes! My research has enlightened someone! This feels good.

I have a crazy hypothesis! Ok, I noticed that 45% of what Romney says is less than half true and exactly 45% of what Obama says is more than half true! Why is this? Well, the obvious answer is Romney is a liar and Obama is a saint, but I am curious if there is a natural skew here. Is it possible since Romney is coming from an attacking standpoint and Obama a defensive they are mostly talking about current policy and that Romney is finding evidence to support his arguement that doesn't tell the whole truth but isn't necessarily a lie, and the Obama is finding evidence to support his that isn't necessarily the whole truth? Would the statistics flip flop if the shoe were on the other foot, and Romney were the incumbent? This is why this blog is awesome b/c we really don't have any past numbers to base this theroy on.


So you're asking if there's any evidence that Romney is less truthful because he is on the attack, and has more at stake. My own analyses can't say much about that (yet), but there's (very weak) evidence refuting your hypothesis. For example, Romney may have been more truthful during the debates, when he had more to prove.

And c'mon man! You can't beat yourself up for the lack of data, and that Politifact is the only source. You gotta use what you got. The fact that there is a Politifact and dude crunching the numbers out there... maybe we are getting more scientific in our politics. This is a good thing! We could start looking at all kinds of things. Like, if Romney is out there spitting out facts at a rating of .35 and all the sudden states fact in the debate at .44 is this why he won the debate in the minds of so many? 


Sadly, I think the truth has less to do with why politicians win or lose political debates that we'd like. But yeah, maybe people subconsciously digest some of the fact checking that goes on after the debates. Also, there is some evidence that people can subconsciously detect lies. But I'm just waving my hands at this point. Still, interesting stuff.


Is this why we see him at times in the nominations "I'll bet you $10,000" as a cartoon character and now as the next leader of the free world based on the polls? Because we hear things we're not that sure of and then BAM on public tv we see this guy's not that far out there and Obama is just hanging around .52, well that's "Obama" big deal. 


Now there you have something. Because Obama is not really as honest as people think he is, and because there is a lot of uncertainty about his honesty, maybe it is one of the things that left him open. Or maybe Romney just got lucky, as the town hall meeting results suggest!


And people all around me are saying I'm kinda liking Romney now or he won the debate. And I say why? They can't articulate it. You don't just feel something, we are perceiving machines wheter or not we can articulate it. Maybe, your approach to understanding politics is getting us a little closer. Thanks again it's really cool and definitely a new idea. Which both sides in Washington could use a few of!


I agree. I hope that Malark-O-Meter catches on.
 
 
Yesterday, I decided I would no longer analyze the average bullpucky of a group, but instead would analyze the collated report cards of a group. 

I just changed my mind again on that. I'm going to do both! Here's why.

If I simulate the probability distribution of average bullpucky, I treat evey member of a group the same. That answers an important question, especially for the people on a presidential ballot ticket. How truthful are they, on average? An analysis of collated report cards asks what the average truthfulness of the statements made by a group are. That's also an important question.

So from here on out, there are two aggregate measures of bullpucky. One measures the average truthfulness of people, and the other the average truthfulness of the statements a group makes. One will be called collated bullpucky. The other will be called average bullpucky.

Here's a rhetorical stats question. If you get it right, I'll think you're nifty. The question: which of these measures will we usually measure as more uncertain?

*sigh* Now I have to edit some of my methods web pages and 
 
 
Below is a plot of the simulated probability distributions of Truth-O-Meter-based bullpucky scores (edit: as of October 15th, 2012) for the presidential and vice presidential candidates. The presidential candidates are in lighter shades, vice presidential candidates in darker shades. I think you can figure out the color scheme. Higher values mean you're more full of bullpucky.
Nobody's overflowing with bullpucky, but nobody's free of it. There are some clear differences between the two parties. And Biden looks like he's more full of it than Obama. There are fewer rated statements for Ryan and Biden compared to Biden and Obama, and Obama has the most statements rated of all. That explains the tight distribution of his bullpucky scores. 

Is it PolitiFact's alleged liberal bias, or is the gap between the two tickets real? Or, a more precise way to ask the question, what are the relative strengths of the effects of rater bias and real differences? Those are the kinds of questions I want to answer at Malark-O-Meter. I'm just getting started.
 
 
This whole thing began with a series of blog posts during the 2012 presidential election. Let's see where it goes from here.
 

    about

    Malark-O-blog published news and commentary about the statistical analysis of the comparative truthfulness of the 2012 presidential and vice presidential candidates. It has since closed down while its author makes bigger plans.

    author

    Brash Equilibrium is an evolutionary anthropologist and writer. His real name is Benjamin Chabot-Hanowell. His wife calls him Babe. His daughter calls him Papa.

    what is malarkey?

    It's a polite word for bullshit. Here, it's a measure of falsehood. 0 means you're truthful on average. 100 means you're 100% full of malarkey. Details.

    what is simulated malarkey?

    Fact checkers only rate a small sample of the statements that politicians make. How uncertain are we about the real truthfulness of politicians? To find out, treat fact checker report cards like an experiment, and use random number generators to repeat that experiment a lot of times to see all the possible outcomes. Details.

    malark-O-glimpse

    Can you tell the difference between the 2012 presidential election tickets from just a glimpse at their simulated malarkey score distributions?

    Picture
    dark = pres, light = vp
    (Click for larger image.)

    fuzzy portraits of malarkey

    Simulated distributions of malarkey for each 2012 presidential candidate with 95% confidence interval on either side of the simulated average malarkey score. White line at half truthful. (Rounded to nearest whole number.)

    Picture
    (Click for larger image.)
    • 87% certain Obama is less than half full of malarkey.
    • 100% certain Romney is more than half full of malarkey.
    • 66% certain Biden is more than half full of malarkey.
    • 70% certain Ryan is more than half full of malarkey.
    (Probabilities rounded to nearest percent.)

    fuzzy portraits of ticket malarkey

    Simulated distributions of collated and average malarkey for each 2012 presidential election ticket, with 95% confidence interval labeled on either side of the simulated malarkey score. White line at half truthful. (Rounded to nearest whole number.)

    malarkometer fuzzy ticket portraits 2012-10-16 2012 election
    (Click for larger image.)
    • 81% certain Obama/Biden's collective statements are less than half full of malarkey.
    • 100% certain Romney/Ryan's collective statements are more than half full of malarkey.
    • 51% certain the Democratic candidates are less than half full of malarkey.
    • 97% certain the Republican candidates are on average more than half full of malarkey.
    • 95% certain the candidates' statements are on average more than half full of malarkey.
    • 93% certain the candidates themselves are on average more than half full of malarkey.
    (Probabilities rounded to nearest percent.)

    Comparisons

    Simulated probability distributions of the difference the malarkey scores of one 2012 presidential candidate or party and another, with 95% confidence interval labeled on either side of simulated mean malarkey. Blue bars are when Democrats spew more malarkey, red when Republicans do. White line and purple bar at equal malarkey. (Rounded to nearest hundredth.)

    Picture
    (Click for larger image.)
    • 100% certain Romney spews more malarkey than Obama.
    • 55% certain Ryan spews more malarkey than Biden.
    • 100% certain Romney/Ryan collectively spew more malarkey than Obama/Biden.
    • 94% certain the Republican candidates spew more malarkey on average than the Democratic candidates.
    (Probabilities rounded to nearest percent.)

    2012 prez debates

    presidential debates

    Simulated probability distribution of the malarkey spewed by individual 2012 presidential candidates during debates, with 95% confidence interval labeled on either side of simulated mean malarkey. White line at half truthful. (Rounded to nearest whole number.)

    Picture
    (Click for larger image.)
    • 66% certain Obama was more than half full of malarkey during the 1st debate.
    • 81% certain Obama was less than half full of malarkey during the 2nd debate.
    • 60% certain Obama was less than half full of malarkey during the 3rd debate.
    (Probabilities rounded to nearest percent.)

    Picture
    (Click for larger image.)
    • 78% certain Romney was more than half full of malarkey during the 1st debate.
    • 80% certain Romney was less than half full of malarkey during the 2nd debate.
    • 66% certain Romney was more than half full of malarkey during the 3rd debate.
    (Probabilities rounded to nearest percent.)

    aggregate 2012 prez debate

    Distributions of malarkey for collated 2012 presidential debate report cards and the average presidential debate malarkey score.
    Picture
    (Click for larger image.)
    • 68% certain Obama's collective debate statements were less than half full of malarkey.
    • 68% certain Obama was less than half full of malarkey during the average debate.
    • 67% certain Romney's collective debate statements were more than half full of malarkey.
    • 57% certain Romney was more than half full of malarkey during the average debate.
     (Probabilities rounded to nearest percent.)

    2012 vice presidential debate

    Picture
    (Click for larger image.)
    • 60% certain Biden was less than half full of malarkey during the vice presidential debate.
    • 89% certain Ryan was more than half full of malarkey during the vice presidential debate.
    (Probabilities rounded to nearest percent.)

    overall 2012 debate performance

    Malarkey score from collated report card comprising all debates, and malarkey score averaged over candidates on each party's ticket.
    Picture
    (Click for larger image.)
    • 72% certain Obama/Biden's collective statements during the debates were less than half full of malarkey.
    • 67% certain the average Democratic ticket member was less than half full of malarkey during the debates.
    • 87% certain Romney/Ryan's collective statements during the debates were more than half full of malarkey.
    • 88% certain the average Republican ticket member was more than half full of malarkey during the debates.

    (Probabilities rounded to nearest percent.)

    2012 debate self comparisons

    Simulated probability distributions of the difference in malarkey that a 2012 presidential candidate spews normally compared to how much they spewed during a debate (or aggregate debate), with 95% confidence interval labeled on either side of the simulated mean difference. Light bars mean less malarkey was spewed during the debate than usual. Dark bars less. White bar at equal malarkey. (Rounded to nearest hundredth.)

    individual 2012 presidential debates

    Picture
    (Click for larger image.)
    • 80% certain Obama spewed more malarkey during the 1st debate than he usually does.
    • 84% certain Obama spewed less malarkey during the 2nd debate than he usually does.
    • 52% certain Obama spewed more malarkey during the 3rd debate than he usually does.
    Picture
    (Click for larger image.)
    • 51% certain Romney spewed more malarkey during the 1st debate than he usually does.
    • 98% certain Romney spewed less malarkey during the 2nd debate than he usually does.
    • 68% certain Romney spewed less malarkey during the 3rd debate than he usually does.

    (Probabilities rounded to nearest percent.)

    aggregate 2012 presidential debate

    Picture
    (Click for larger image.)
    • 58% certain Obama's statements during the debates were more full of malarkey than they usually are.
    • 56% certain Obama spewed more malarkey than he usually does during the average debate.
    • 73% certain Romney's statements during the debates were less full of malarkey than they usually are.
    • 86% certain Romney spewed less malarkey than he usually does during the average debate.

    (Probabilities rounded to nearest percent.)

    vice presidential debate

    Picture
    (Click for larger image.)
    • 70% certain Biden spewed less malarkey during the vice presidential debate than he usually does.
    • 86% certain Ryan spewed more malarkey during the vice presdiential debate than he usually does.

    (Probabilities rounded to nearest percent.)

    2012 opponent comparisons

    Simulated probability distributions of the difference in malarkey between the Republican candidate and the Democratic candidate during a debate, with 95% confidence interval labeled on either side of simulated mean comparison. Blue bars are when Democrats spew more malarkey, red when Republicans do. White bar at equal malarkey. (Rounded to nearest hundredth.)

    individual 2012 presidential debates

    Picture
    (Click for larger image.)
    • 60% certain Romney spewed more malarkey during the 1st debate than Obama.
    • 49% certain Romney spewed more malarkey during the 2nd debate than Obama.
    • 72% certain Romney spewed more malarkey during the 3rd debate than Obama.

    (Probabilities rounded to nearest percent.)

    aggregate 2012 presidential debate

    Picture
    (Click for larger image.)
    • 74% certain Romney's statements during the debates were more full of malarkey than Obama's.
    • 67% certain Romney was more full of malarkey than Obama during the average debate.

    (Probabilities rounded to nearest percent.)

    vice presidential debate

    • 92% certain Ryan spewed more malarkey than Biden during the vice presidential debate.

    (Probabilities rounded to nearest percent.)

    overall 2012 debate comparison

    Party comparison of 2012 presidential ticket members' collective and individual average malarkey scores during debates.
    • 88% certain that Republican ticket members' collective statements were more full of malarkey than Democratic ticket members'.
    • 86% certain that the average Republican candidate spewed more malarkey during the average debate than the average Democratic candidate.

    (Probabilities rounded to nearest percent.)

    observe & report

    Below are the observed malarkey scores and comparisons form the  malarkey scores of the 2012 presidential candidates.

    2012 prez candidates

    Truth-O-Meter only (observed)

    candidate malarkey
    Obama 44
    Biden 48
    Romney 55
    Ryan 58

    The Fact Checker only (observed)

    candidate malarkey
    Obama 53
    Biden 58
    Romney 60
    Ryan 47

    Averaged over fact checkers

    candidate malarkey
    Obama 48
    Biden 53
    Romney 58
    Ryan 52

    2012 Red prez vs. Blue prez

    Collated bullpucky

    ticket malarkey
    Obama/Biden 46
    Romney/Ryan 56

    Average bullpucky

    ticket malarkey
    Obama/Biden 48
    Romney/Ryan 58

    2012 prez debates

    1st presidential debate

    opponent malarkey
    Romney 61
    Obama 56

    2nd presidential debate (town hall)

    opponent malarkey
    Romney 31
    Obama 33

    3rd presidential debate

    opponent malarkey
    Romney 57
    Obama 46

    collated presidential debates

    opponent malarkey
    Romney 54
    Obama 46

    average presidential debate

    opponent malarkey
    Romney 61
    Obama 56

    vice presidential debate

    opponent malarkey
    Ryan 68
    Biden 44

    collated debates overall

    ticket malarkey
    Romney/Ryan 57
    Obama/Biden 46

    average debate overall

    ticket malarkey
    Romney/Ryan 61
    Obama/Biden 56

    the raw deal

    You've come this far. Why not just check out the raw data Maslark-O-Meter is using? I promise you: it is as riveting as a phone book.

    archives

    June 2013
    May 2013
    April 2013
    January 2013
    December 2012
    November 2012
    October 2012

    malark-O-dex

    All
    2008 Election
    2012 Election
    Average Malarkey
    Bias
    Brainstorm
    Brier Score
    Bullpucky
    Caveats
    Closure
    Collated Malarkey
    Conversations
    Dan Shultz
    Darryl Holman
    Debates
    Drew Linzer
    Election Forecasting
    Equivalence
    Fact Checking Industry
    Fallacy Checking
    Foreign Policy
    Fuzzy Portraits
    Gerrymandering
    Incumbents Vs. Challengers
    Information Theory
    Kathleen Hall Jamieson
    Launch
    Logical Fallacies
    Longitudinal Study
    Malarkey
    Marco Rubio
    Meta Analysis
    Methods Changes
    Misleading
    Model Averaging
    Nate Silver
    Origins
    Pants On Fire
    Politifactbias.com
    Poo Flinging
    Presidential Election
    Ratios Vs Differences
    Redistricting
    Red Vs. Blue
    Root Mean Squared Error
    Sam Wang
    Science Literacy
    Short Fiction
    Simon Jackman
    Small Multiples
    Stomach Parasite
    The Future
    The Past
    To Do
    Truth Goggles
    Truth O Meter
    Truth O Meter