3

This footnote by Efron appears in the chapter on false discovery rate control:

I am trying to avoid the term "significant" for the rejected cases as dubious terminology even in single-case testing, and worse in the false discovery rate context, preferring instead "interesting". --Bradley Efron "Large Scale Inference" 2010, p.47

What are the arguments pro and con adopting this change in nomenclature? Is it significant that such a prominent statistician is avoiding "significance"?

Biomedicine and healthcare in particular are domains where quantitative evidence is largely based on association studies, whether RCTs or observational. "Links" between therapies and outcomes, or biomarkers and disease states, depend on hypothesis testing framework.

Could adopting "interesting" on a large scale induce reconsideration of the strength of evidence associated with of such linkages, even if p-value or z-scores still quantify the results?

alancalvitti
  • 141
  • 6
  • @gung, thx, edited – alancalvitti Apr 25 '12 at 21:51
  • This is just a comment, b/c I don't have much substantive to add; but if we think of this question abstractly as 'alternative approaches to ( /ways to think about) hypothesis testing that might address some of the typical problems', then you might find [these two](http://stats.stackexchange.com/questions/23617/) previous [CV questions](http://stats.stackexchange.com/questions/24061/) worth reading. – gung - Reinstate Monica Apr 26 '12 at 03:26

1 Answers1

2

Like many statistical terms, "significant" has a very precise meaning that many non-expert users misinterpret. Statisticians use it to mean statistical significance (i.e. results beyond chance, at some alpha) while non-statisticians use it to mean practical significance (or practically interesting, if you prefer). Statistical significance is neither a necessary or sufficient condition for practical significance; meaning results can be statistically not practically significant, or practically but not statistically significant. Is it significant that Efron wants to drop the term? Not statistically significant, as all (good) statisticians are aware of the problem. But maybe it's practically significant, as Efron reaches a big audience, some of whom might not otherwise listen to these concerns.

I'm afraid I don't think substituting "interesting" helps the underlying problem. If (two-sided) p-values are used to measure evidence, then we're in foundational trouble; see Schervish 1996 (American Statistician) for a discussion of how two-sided p-values, alone, don't behave like any reasonable measure of evidence. While this difficulty doesn't rule out getting sensible results from p-values in some situations, it's generally more helpful to work on the scale of the data measurements themselves, not their transformation to p's or z's. Efron works pretty hard to stick to p's and z's.

guest
  • 2,381
  • 14
  • 11
  • Agreed and thanks for the ref to Schervish. However I attend biomedical informatics seminars (UCSD) where p-values are often the only quantitative results presented. My background is in engineering where even in stochastic contexts, p-values are basically unheard of. Similarly significance and p-values are absent from the majority of books in the QA-QD section of the library. So I agreed that sensible results should be based on measurements whose units are universally understood. – alancalvitti Apr 26 '12 at 14:19