TL;DR ASA suggests that researchers should report effect sizes, confidence intervals, and other measures of uncertainty alongside p-values.
For decades, p-values have been the go-to statistical tool for determining statistical significance in scientific research. However, recent criticisms of p-values have led many researchers to question their usefulness and accuracy. In response to these criticisms, the American Statistical Association (ASA) released a statement on statistical significance and p-values, calling for rethinking their role in scientific research.
The ASA statement highlights several limitations of p-values, including their susceptibility to misinterpretation, dependence on arbitrary thresholds, and inability to measure the strength of evidence. To supplement or replace p-values, the ASA suggests that researchers should report effect sizes, confidence intervals, and other measures of uncertainty alongside p-values.
Frank Harrell’s blog post, “The P-Value Litany,” echoes the concerns raised by the ASA statement and calls for a move beyond p-values in scientific research. Harrell points out that p-values are affected by sample size and study design and do not provide information about effect size or the uncertainty of estimates. Instead, he suggests that researchers use confidence intervals and Bayesian statistics, which give a more complete picture of the data and are not subject to the same limitations as p-values.
In addition to supplementing or replacing p-values, the ASA statement and Harrell emphasize the importance of considering the clinical or practical significance of findings rather than relying solely on statistical significance. This involves evaluating effect sizes’ magnitude and relevance to real-world settings.
The practical lesson is simple: p-values should not carry the whole argument. Report effect sizes, uncertainty intervals and subject-matter relevance so readers can judge both the statistical and practical meaning of the result.
Reference: