Bernoulli's Fallacy
In statistics courses in undergraduate and graduate, as well as corporate statistical training, I was repeatedly instructed in statistical tests and null hypothesis. I was never given any hint that I was only being taught only the frequentist school of statistics and completely neglecting the other - Bayesean1. It appears that I’m old enough now to have some of my traditional education be, if not proven wrong, at least inadequate. The book I recently finished Bernoulli’s Fallacy, Statistical Illogic and the Crisis of Modern Science does a good job of giving the history behind why Bayes was neglected through much of the twentieth century and the flaws in frequentist statistics, and further explains why that is such a mistake.
The author starts the book with the fundamental idea that you have to work your way through to understand the flaw in frequentist statistics.
The probability of the hypothesis given the data is not equal to the probability of the data given the hypothesis.
This counter-intuitive statement follows directly from Bayes theorem:
P(H|D) = P(D|H) * P(H)/P(D)
which follows from a simple algebraic manipulation of the following identity2:
P(H and D) = P(H|D)*P(D) = P(D|H)*P(H)
All of frequentist probability focuses on the what the probability of the data is given the hypothesis. If the data is very unlikely, one rejects the null hypothesis. But Bayes tells you you must also consider the prior probability P(H).
Going further requires getting into the epistemology of probability. Does probability have meaning for one time events? Frequentists say no, but Bayes’s followers are more open to the idea.
After reading this book I’m going to be much more skeptical anytime I hear phrase statistically significant and attempt to incorporate my prior and Bayes when working with probabilities.