P(Type I error | significant)

Probability that a reported significant result reflects no real effect.

Positive Predictive Value (PPV)

Complementary view: probability that a reported significant result reflects a real effect.

Each cell represents one of 1000 hypothetical studies in the field under the parameters chosen on the left. The colour encodes two pieces of information at once: the hue tells you whether the study reached the correct conclusion (green) or an incorrect one (red), and the shade tells you whether the study was reported as statistically significant (dark) or not (light).

  • Dark green — true positive. The study reached statistical significance and a real effect underlies the data. The published finding is correct.
  • Light green — true negative. The study did not reach significance, and there is no real effect. The study typically never reaches the literature, but in this case that is the right outcome.
  • Dark red — false positive (Type I error). The study reached statistical significance, but no real effect underlies the data. The finding gets published and, to a reader scanning the journal, looks indistinguishable from a dark-green cell.
  • Light red — false negative (Type II error). A real effect exists, but the study was underpowered or unlucky and missed it. The result ends up in the file drawer.

Dashed line marks the current value of the swept parameter.

Among all hypotheses tested in a field, let π = P(H1 true) be the prior probability that any given hypothesis reflects a real effect. The honest 2×2 proportions are:

TP = π(1 − β)        FN = πβ
FP = (1 − π)α        TN = (1 − π)(1 − α)

With reporting bias u, a fraction u of the would-be negatives is reclassified into the ‘significant’ pile:

PPV = [π(1−β) + u·πβ] / [π(1−β) + u·πβ + (1−π)α + u·(1−π)(1−α)]

P(Type I error | significant) = 1 − PPV.

Reference: Ioannidis JPA (2005), PLoS Medicine 2(8):e124.