7

In a comment to Jaap's answer to this question about reporting tiny p-values, amoeba says "Once the $p$-value is below 0.0001 or something ... it's probably more meaningful to look at $z$-value than at $p$-value," for example the 5$\sigma$ convention in particle physics. There was agreement that this was right. Why is it right? Why are $z$-values a more meaningful way to describe data that gives very low $p$-values? What does "more meaningful" mean?

Quoting Glen_b's answer to why very low $p$-values aren't that meaningful:

But statistical meaning will have been lost far earlier. Note that $p$-values depend on assumptions, and the further out into the extreme tail you go the more heavily the true $p$-value (rather than the nominal value we calculate) will be affected by the mistaken assumptions, in some cases even when they're only a little bit wrong. Since the assumptions are simply not going to be all exactly satisfied, middling $p$-values may be reasonably accurate (in terms of relative accuracy, perhaps only out by a modest fraction), but extremely tiny $p$-values may be out by many orders of magnitude.

Why doesn't similar logic apply to really large $z$-values? Don't they have assumptions that can be violated?

Related questions: If this is the case, why not just always use $z$-values? Are $p$-values "more meaningful" than $z$-values when the $p$-value is high?

R Greg Stacey
  • 2,202
  • 2
  • 15
  • 30
  • 5
    $z$-value is just an estimate of some parameter divided by its standard error. No assumptions are needed to compute it, assumptions would be needed to interpret it. The p-value provides such an interpretation. When we say that $z=1.96$ corresponds to $p=.05$ we are making the normality assumption. If we obtain $z=100$ then as Glen_b explains, it does not make sense to convert it into the exact p-value anymore. However, one might want to convey the message that the p-value would be *freaking small*. Reporting $z=100$ seems like a good way to convey just that. That's what I meant. – amoeba Jun 28 '17 at 22:57
  • @amoeba, why not develop that into an official answer? – gung - Reinstate Monica Jun 29 '17 at 18:08
  • @amoeba Thanks. I agree about writing it as an official answer. From what I can tell you answered my first question. Also a few more sentences would be really helpful, e.g. my last two questions. – R Greg Stacey Jun 29 '17 at 19:12

1 Answers1

3

A lower $p$-value does not indicate the assumptions were violated. Some really low $p$-values like $2.22\times10^{-16}$ just indicate the limit of the machine, something called machine epsilon. Once a number gets that low, the machine can't go any lower, so spits it out or zero.

The reason one could report the $z$ at that point is $z$ can be interpreted meaningfully within certain contexts. A small $p$ is just really really really small. Additionally, at that point, it is irrelevant. There is a meaningful difference between $p_a=.35$ and $p_b=.09$. However, there is little substantive difference between $p_1=.0000000001$ and $p_2=.0000000000000000001$, even though $\frac{p_1}{p_2}$ is many times larger than $\frac{p_a}{p_b}$.

Finally, the author claims that as $p$-values get really small, failure to meet statistical assumptions have more effects on the magnitude of the $p$-value. We almost never fully meet assumptions of statistical tests, but if we have a $p$-value of .04, and we did a decent job of meeting the assumptions of the test, its magnitude could vary between .02 and say, .06. However, if we have a $p$-value like $p_2$, the author is saying it could be as different as $p_1$ in reality. So at that level, the number you get is not reliable - it is a false precision.

This does not apply to the $z$ much as the change in $p$, as large as from $p_1$ to $p_2$, is just a 3 point change in $z$ from about 6 to 9. And on $z$, if we are concerned with the uncertainty about it, we can always report its confidence interval. Of course, we could do the same for a tiny $p$ but that would be very unconventional and barely interpretable.

Heteroskedastic Jim
  • 4,567
  • 1
  • 10
  • 32
  • I quoted Glen_b's answer for why very low $p$-values aren't that meaningful. – R Greg Stacey Jun 28 '17 at 22:32
  • I'll modify my answer. – Heteroskedastic Jim Jun 28 '17 at 22:34
  • Are you just saying that z is more human readable for extreme values? I disagree that "a small p is just really really small" and "there is little substantive difference" between p1 and p2. They're difficult to read, but isn't there a "substantive difference" between 10^-9 and 10^-19? – R Greg Stacey Jun 29 '17 at 19:07
  • Partly that z is human interpretable, but also because you are in error territory, false precision. I will guess that Glen_b means that we know from simulation that as you reduce the nominal alpha level, violations of assumptions matter a lot more. So at .05, maybe not so much but at 10^-19, they matter too much for such p-values to be meaningful in any way. They are really low, as to their true magnitude? We cannot be certain. – Heteroskedastic Jim Jun 29 '17 at 19:46