4

Consider the following definition of statistical significance:

Statistical significance is a characteristic of a statistic viewed in light of an (implicit or explicit) null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region or the acceptance region defined by the null hypothesis and the significance level. The statistic is then statistically significant or not statistically significant, respectively.

Is the definition valid, or are there any problems with it?

(It is quite different from e.g. our tag description for statistical significance.)

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • 1
    The tag wiki could use some clarification, because it is really trying to describe the p-value. Although closely related to significance, it's not the same thing. – whuber Apr 15 '21 at 14:00
  • 1
    @whuber, indeed, this is what I thought. What about replacing it with what I have in the post, maybe with a tweak or two if needed? – Richard Hardy Apr 15 '21 at 14:48

1 Answers1

0

The tag description has been changed to the following:

Statistical significance is a characteristic of a statistic viewed in light of an (implicit or explicit) null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region or the acceptance region defined by the null hypothesis and the significance level. The statistic is then statistically significant or not statistically significant, respectively.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • 1
    But let's recognize that "statistically significant" means almost nothing. It is an arbitrary construct that invites users to make the "absence of evidence is not evidence of absence" error among other things. In light of the fact that the rejection region is an arbitrary construct not tied to decision theory, I'd like to see a definition that at least mentions the word "arbitrary" and points to the ASA statement on p-values. – Frank Harrell Jun 06 '21 at 11:49
  • 2
    @FrankHarrell Why do you say the rejection region is unrelated to decision theory? – Dave Jun 06 '21 at 11:51
  • The Bayes optimum decision is the decision that optimizes the expected utility/loss/cost function. P-values have no relationship to that since they condition on the decision ($H_0$) you are trying to make in the first place. Expected utility is the integral of the posterior probability convolved with the utility whereas the P-value is a probability about _data_ not about $\theta$. – Frank Harrell Jun 06 '21 at 11:55
  • 1
    I got the impression the rejection region is not all that arbitrary: e.g. https://stats.stackexchange.com/questions/447510, https://stats.stackexchange.com/questions/483685. – Richard Hardy Jun 06 '21 at 13:54
  • 1
    In this situation I have, reluctantly, to disagree with Prof. Harrell, who seems to adopt a Bayesian perspective that is a little too extreme. The decision-theoretic foundations of statistical significance have been rigorously developed by Wald, Kiefer, and others. As they clearly showed, the rejection region is far from "arbitrary." Statistical significance (and the p-value) has a principled, logical meaning. The problems with p-values stem from misunderstandings, abuse, and inherent limitations rather than arbitrariness in the theory or their construction. – whuber Jun 06 '21 at 17:00