Mark “Language Log” Liberman is taking Steven D. “Freakonomics” Levitt to task for either misunderstanding the language of statistics, or the underlying statistics theory itself.
In a blog post, “Medicine and Statistics Don’t Mix,” Levitt tells the story of friends of his who spent $5,000 on Preimplantation Genetic Diagnosis (PGD) — a test for genetic viability of embryos — during fertility treatment. The test came back positive for the possibility of birth defects. The friends decided to go ahead anyway–after spending another $5,000 and getting a positive result–and their babies were born without defect. Levitt’s point is: “I never trust statistics I get from people in the field of medicine, ever.”
Liberman rightly criticizes Levitt for confusing ‘false positive rate’ with the probability that a condition holds given a test indicating the condition, or “positive predictive value.” Some of the direct commentators on Levitt’s blog post have mentioned this error as well.
But no one seems to ask the basic question in the current case: what is the positive predictive value of PGD? I’m not exactly sure, but this PubMed article seems to indicate it’s in the 90-95% range, which (given the hearsay chain: doctors tell friends, who tell Levitt, who tell us) seems roughly consistent with the 10% ‘false positive rate’ Levitt cites. So perhaps the problem is a terminological one — when Levitt said ‘false positive rate,’ he really meant ‘positive predictive value.’
Anyway, I think the morals of the story are:
- If you have a medical test and it shows positive, you should ask your doctor what the “positive predictive value” of the test is. These seems to be the term of art in the medical field. Although there will be lots of assumptions built into this, this will tell you the probability that you have the problem given the positive test results.
- You should also ask what the base rate of the condition is — the probability that you have the condition before the test is done.
- Before spending money on an expensive test, it might be worth asking: what difference will it make if I have the additional information? In the case of Levitt’s friends, they spend $10,000 (out of their own pocket, I think) for tests whose results they ignored.
Dear statistical and medical friends: tell me if and how I’m wrong.