Fair warning, this post includes math.

Ok, now for the two of you who have stuck around... the reason I bring this up is because of the ads I've seen for a new TV show. Now, normally I barely see commercials since I bought TiVo (aside from the personal computer, my favorite gadget in the world). But moving into a new home means that TiVo's not been hooked up, and if I want a respite from unpacking, or to see the games (fucking OSU -- seriously, people, come on), I'm stuck with the ads.

The newest game show to hit prime time is something called "The Moment of Truth". If you don't click through to read about it, and I don't blame anyone who doesn't, the idea is this: sit someone in a chair, strap them to a lie detector (polygraph) and ask embarrassing questions. You win money if you answer "honestly". Why do I put that in scare quotes? Well, that brings me to the math. Almost.

Polygraph machines have been around a long time, and are a key feature in things like police investigations and security clearance investigations. Which means they feature prominently in the prosecution of justice safeguarding of secrets. But do they work?

Not the way they'll tell you they work. Here's the problem: detecting a lie isn't easy, and the test relies on subtle physical changes that are believed to be cued when you lie. That they may be cued when you are stressed (in times when, for instance, you may be being questioned about a crime or are trying to get a clearance to get a job) is an open question, but we'll skip over it for the sake of simplicity. When asked, polygraph examiners will cite how "accurate" the test is as a sign of how well it works. This is bullshit. It may be great interrogation, but it's mathematically worthless. The problem is that a test like this has not only a chance of being right, but also a chance of being wrong.

Bergeron will likely know about an analogous situation: You go to the doctor and have a test done to see if you have Terrible Horrible Impotence Nurturing Genetic Enzymes. The test for THINGE says it is 95% accurate. So, your THINGE test comes back positive. What then, is the probability that you have THINGE?

Did you say 95%? Well, I'm guessing most people go to the point of "Well, I would, but this seems like a trick question." Give yourself points if you did. Because it is NOT necessarily 95%. We need to know more things. That 95% rate is essentially the true-positive rate. That is, 95% of the time it will give a positive reading in the presence of someone who has THINGE. But it will also have a false-positive rate. In this case, make it 5%. And we have to know the underlying rate of the disease in the population. Let's make it easy and pick .10; that is, 1 in 10 people in the entire population have THINGE. So, you got a positive test back--what is the probability you have THINGE?

It's this: The chance that the test comes back positive GIVEN that you have THINGE times the chance that you have THINGE, all divided by that plus the opposite (a positive test result given you don't have THINGE times the chance that you don't have THINGE.

Let's do the best part of math and shorten all that:

• P(p|T) is the probability of a positive test result given you have THINGE, or, the "test accuracy"


• P(T) is the probability you have THINGE, or the .10 rate


• P(C) is the probability you are clear of THINGE, or 1-P(T) = .9


• P(p|C) is the probability of a positive test given you DON'T have THINGE, or, the "false positive" of .05

Ok, then: P(p|T) x P(T) is .95 x .10 = .095
And: P(p|T) x P(T) + P(p|C) x P(C) = .095 + (.05 x .9) = .14.
Finally: .095/.14 = .6786.(*)

With a positive test result, on a test that is 95% accurate, you have a 68% chance of having THINGE. Not good odds for you, but much better than 95%, eh? Which is the reason doctors have multiple tests and look for multiple possible indicators for a condition.

Back to the main point: the exact same issue holds for polygraphs. Proponents claim they are "close to 100% accurate", while critics say they are only around 80% accurate. This is a huge gap, though it might not look like it.

Let's say you're being screened for a security clearance. They want to know if you've murdered anyone. How confident can they be that they weed out a murderer? Well, just go through the calculation again. This time, we'll use the US average rate for homicide, and we'll use the 2005 18-24 grouping, which is the highest rate: 26.5 out of every 100,000, or .0265% . And, lets say the test is really good: 95% accurate, with a 5% false positive rate. Finally, needless to say, you answer no. (We won't say whether or not you did yet.)

Doing the calculation as above, we get that the probability that you murdered someone given that the test is positive (show's you are lying by answering "no" to the question of whether you have ever committed homicide), is around 34%. Barely better than one in three, assuming that nothing other than you lying affects the results of the test.

In the case of the TV show, things are even worse. How do you get an underlying distribution of people who are racist, who have thought about their boss sexually, or how do you judge if someone seriously considered divorce or if they just had a shitty day?

In the end, this is just idiot people doing anything for money. And it's their choice, so good for them. The real disservice is that even the people behind the show, even the people who administer the test, may have absolutely no idea how bad their test actually is.

(*) For the interested, this is simply Bayes' Theorem.