P-Fishing with Unlikely Hypotheses

My old friend Betsy Paluck tweeted this classic post by my old teacher Don Green last night.

The post makes a clear point really well: “the nominal results of an experiment are robbed of their persuasive value if readers suspect that the findings were obtained through fishing.” (See my own discussion of fishing here.)

The best thing about this Bayesian perspective is that it requires us to be precise about subjective beliefs. In this example, we have the prior probability that a hypothesis is true before we see the data (Pr(H)), the probability that one would see a significant result result given that the hypothesis is true (Pr(E|H)), and the probability that we would see that result given the hypothesis is false (Pr(E|~H)). Using Bayes rule, we obtain a posterior probability, given the results, that the hypothesis is true: (Pr(H|E)). The chosen example begins with the standard “equal probability” belief that a hypothesis is true: Pr(H) = .5 or a 50/50 chance. And to illustrate p-fishing we see that there’s a 75% chance of getting a significant result: Pr(E|H) = .75.

Well, what if we varied those quantities a bit? What if instead of increasing Pr(E|H) from .05 with no fishing to .75 with fishing, we chose some intermediate values? And what if we our subjective prior beliefs were more skeptical than the 50/50 scenario? Here is what we would get:

bayes example

The × in the figure corresponds to Green’s example: Pr(E|H) = .75 and Pr(H) = .5. But if we are sufficiently skeptical of the hypothesis to begin with, and the possibility of p-fishing is a bit more moderate, then we might still find reason to believe the results. (The dashed horizontal line shows us the magic p=.05 boundary where all results are true and publishable (kidding!)). The intuition is that when we are more skeptical about a claim to begin with, evidence that is inconsistent with our priors has a large effect even when we account for the probability that the signal sent by that evidence is noisy.

Does this imply that if you are testing really crazy hypotheses you don’t have to pre-register? Not at all. Just that the lessons that we draw from Bayes rule depend on our priors about the hypotheses that we test and the extent of p-fishing. And yes, if we decrease the values of Pr(E|~H), set here at 1, then Pr(H|E) will jump up again.