Full Terms & Conditions of access and use can be found at
https:
amstat.tandfonline.com/action/journalInformation?journalCode=utas20
The American Statistician
ISSN: XXXXXXXXXXPrint XXXXXXXXXXOnline) Journal homepage: https:
amstat.tandfonline.com/loi/utas20
Moving to a World Beyond “p < 0.05”
Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Laza
To cite this article: Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019)
Moving to a World Beyond “p < 0.05”, The American Statistician, 73:sup1, 1-19, DOI:
10.1080/ XXXXXXXXXX1583913
To link to this article: https:
doi.org/10.1080/ XXXXXXXXXX1583913
© 2019 The Author(s). Published by Informa
UK Limited, trading as Taylor & Francis
Group.
Published online: 20 Mar 2019.
Submit your article to this journal
Article views: 76682
View Crossmark data
Citing articles: 8 View citing articles
https:
amstat.tandfonline.com/action/journalInformation?journalCode=utas20
https:
amstat.tandfonline.com/loi/utas20
https:
amstat.tandfonline.com/action/showCitFormats?doi=10.1080/ XXXXXXXXXX1583913
https:
doi.org/10.1080/ XXXXXXXXXX1583913
https:
amstat.tandfonline.com/action/authorSubmission?journalCode=utas20&show=instructions
https:
amstat.tandfonline.com/action/authorSubmission?journalCode=utas20&show=instructions
http:
crossmark.crossref.org/dialog/?doi=10.1080/ XXXXXXXXXX1583913&domain=pdf&date_stamp= XXXXXXXXXX
http:
crossmark.crossref.org/dialog/?doi=10.1080/ XXXXXXXXXX1583913&domain=pdf&date_stamp= XXXXXXXXXX
https:
amstat.tandfonline.com/doi/citedby/10.1080/ XXXXXXXXXX1583913#tabModule
https:
amstat.tandfonline.com/doi/citedby/10.1080/ XXXXXXXXXX1583913#tabModule
THE AMERICAN STATISTICIAN
2019, VOL. 73, NO. S1, 1–19: Editorial
https:
doi.org/10.1080/ XXXXXXXXXX1583913
EDITORIAL
Moving to a World Beyond “p < 0.05”
Some of you exploring this special issue of The American Statis-
tician might be wondering if it’s a scolding from pedantic statis-
ticians lecturing you about what not to do with p-values, without
offering any real ideas of what to do about the very hard problem
of separating signal from noise in data and making decisions
under uncertainty. Fear not. In this issue, thanks to 43 innovative
and thought-provoking papers from forward-looking statisti-
cians, help is on the way.
1. “Don’t” Is Not Enough
There’s not much we can say here about the perils of p-values
and significance testing that hasn’t been said already for decades
(Ziliak and McCloskey 2008; Hu
ard XXXXXXXXXXIf you’re just a
iv-
ing to the debate, here’s a sampling of what not to do:
• Don’t base your conclusions solely on whether an association
or effect was found to be “statistically significant” (i.e., the p-
value passed some a
itrary threshold such as p < 0.05).
• Don’t believe that an association or effect exists just because
it was statistically significant.
• Don’t believe that an association or effect is absent just
ecause it was not statistically significant.
• Don’t believe that your p-value gives the probability that
chance alone produced the observed association or effect o
the probability that your test hypothesis is true.
• Don’t conclude anything about scientific or practical impor-
tance based on statistical significance (or lack thereof).
Don’t. Don’t. Just…don’t. Yes, we talk a lot about don’ts. The ASA
Statement on p-Values and Statistical Significance (Wasserstein
and Lazar 2016) was developed primarily because after decades,
warnings about the don’ts had gone mostly unheeded. The
statement was about what not to do, because there is widespread
agreement about the don’ts.
Knowing what not to do with p-values is indeed necessary,
ut it does not suffice. It is as though statisticians were asking
users of statistics to tear out the beams and struts holding up
the edifice of modern scientific research without offering solid
construction materials to replace them. Pointing out old, rotting
timbers was a good start, but now we need more.
Recognizing this, in October 2017, the American Statistical
Association (ASA) held the Symposium on Statistical Infer-
ence, a two-day gathering that laid the foundations for this
special issue of The American Statistician. Authors were explic-
itly instructed to develop papers for the variety of audiences
interested in these topics. If you use statistics in research, busi-
ness, or policymaking but are not a statistician, these articles
were indeed written with YOU in mind. And if you are a
statistician, there is still much here for you as well.
The papers in this issue propose many new ideas, ideas that
in our determination as editors merited publication to enable
oader consideration and debate. The ideas in this editorial are
likewise open to debate. They are our own attempt to distill the
wisdom of the many voices in this issue into an essence of good
statistical practice as we cu
ently see it: some do’s for teaching,
doing research, and informing decisions.
Yet the voices in the 43 papers in this issue do not sing as
one. At times in this editorial and the papers you’ll hear deep
dissonance, the echoes of “statistics wars” still simmering today
(Mayo XXXXXXXXXXAt other times you’ll hear melodies wrapping in a
ich counterpoint that may herald an increasingly harmonious
new era of statistics. To us, these are all the sounds of statistical
inference in the 21st century, the sounds of a world learning to
venture beyond “p < 0.05.”
This is a world where researchers are free to treat “p = 0.051”
and “p = 0.049” as not being categorically different, where
authors no longer find themselves constrained to selectively
publish their results based on a single magic number. In this
world, where studies with “p < 0.05” and studies with “p
0.05” are not automatically in conflict, researchers will see thei
esults more easily replicated—and, even when not, they will
etter understand why. As we venture down this path, we will
egin to see fewer false alarms, fewer overlooked discoveries,
and the development of more customized statistical strategies.
Researchers will be free to communicate all their findings in all
their glorious uncertainty, knowing their work is to be judged
y the quality and effective communication of their science, and
not by their p-values. As “statistical significance” is used less,
statistical thinking will be used more.
The ASA Statement on P-Values and Statistical Significance
started moving us toward this world. As of the date of publi-
cation of this special issue, the statement has been viewed ove
294,000 times and cited over 1700 times—an average of about
11 citations per week since its release. Now we must go further.
That’s what this special issue of The American Statistician sets
out to do.
To get to the do’s, though, we must begin with one more don’t.
© 2019 The Author(s). Published with license by Taylor & Francis Group, LLC.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http:
creativecommons.org/licenses
y-nc-nd/4.0/), which
permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.
https:
doi.org/10.1080/ XXXXXXXXXX1583913
https:
crossmark.crossref.org/dialog/?doi=10.1080/ XXXXXXXXXX1583913&domain=pdf&date_stamp= XXXXXXXXXX
http:
creativecommons.org/licenses
y-nc-nd/4.0
2 EDITORIAL
2. Don’t Say “Statistically Significant”
The ASA Statement on P-Values and Statistical Significance
stopped just short of recommending that declarations of
“statistical significance” be abandoned. We take that step here.
We conclude, based on our review of the articles in this special
issue and the
oader literature, that it is time to stop using
the term “statistically significant” entirely. Nor should variants
such as “significantly different,” “p < 0.05,” and “nonsignificant”
survive, whether expressed in words, by asterisks in a table, o
in some other way.
Regardless of whether it was ever useful, a declaration of
“statistical significance” has today become meaningless. Made
oadly known by Fisher’s use of the phrase (1925), Edgeworth’s
(1885) original intention for statistical significance was simply
as a tool to indicate when a result wa
ants further scrutiny. But
that idea has been i
etrievably lost. Statistical significance was
never meant to imply scientific importance, and the confusion of
the two was decried soon after its widespread use (Boring 1919).
Yet a full century later the confusion persists.
And so the tool has become the tyrant. The problem is not
simply use of the word “significant,” although the statistical and
ordinary language meanings of the word are indeed now hope-
lessly confused (Ghose 2013); the term should be avoided fo
that reason alone. The problem is a larger one, however: using
ight-line rules for justifying scientific claims or conclusions
can lead to e
oneous beliefs and poor decision making (ASA
statement, Principle 3). A label of statistical significance adds
nothing to what is already conveyed by the value of p; in fact,
this dichotomization of p-values makes matters worse.
For example, no p-value can reveal the plausibility, presence,
truth, or importance of an association or effect. Therefore, a
label of statistical significance does not mean or imply that an
association or effect is highly probable, real, true, or important.
Nor does a label of statistical nonsignificance lead to the associ-
ation or effect being improbable, absent, false, or unimportant.
Yet the dichotomization into “significant” and “not significant”
is taken as an imprimatur of authority on these characteristics.
In a world without
ight lines, on the other hand, it becomes
untenable to assert dramatic differences in interpretation from
inconsequential differences in estimates. As Gelman and Stern
(2006) famously observed, the difference between “significant”
and “not significant” is not itself statistically significant.
Furthermore, this false split into “worthy” and “unworthy”
esults leads to the selective reporting and publishing of results
ased on their statistical significance—the so-called “file drawe
problem” (Rosenthal XXXXXXXXXXAnd the dichotomized reporting
problem extends beyond just publication, notes Amrhein,
Trafimow, and Greenland (2019): when authors use p-value
thresholds to select which findings to discuss in their papers,
“their conclusions and what is reported in subsequent news
and reviews will be biased…Such selective attention based on
study outcomes will therefore not only distort the literature but
will slant published descriptions of study results—biasing the
summary descriptions reported to practicing professionals and
the general public.” For the integrity of scientific publishing
and research dissemination, therefore, whether a p-value passes
any a
itrary threshold should not be considered at all when
deciding which results to present or highlight.
To be clear, the problem is not that of having only two
labels. Results should not be trichotomized, or indeed catego-
ized into any number of groups, based on a
itrary p-value
thresholds. Similarly, we need to stop using confidence inter-
vals as another means of dichotomizing (based, on whether a
null value falls within the interval). And, to preclude a reap-
pearance of this problem elsewhere, we must not begin a
i-
trarily categorizing other statistical measures (such as Bayes
factors).
Despite the limitations of p-values (as noted in Princi-
ples 5 and 6 of the ASA statement), however, we are not
ecommending that the calculation and use of continuous
p-values be discontinued. Where p-values are used, they
should be reported as continuous quantities (e.g., p = 0.08).
They should also be described in language stating what
the value means in the scientific context. We believe that
a reasonable prerequisite for reporting any p-value is the
ability to interpret it appropriately. We say more about this in
Section 3.3.
To move forward to a world beyond