12

In network motif algorithms, it seems quite common to return both a p-value and a Z-score for a statistic: "Input network contains X copies of subgraph G". A subgraph is deemed a motif if it satisfies

  • p-value < A,
  • Z-score > B and
  • X > C, for some user-defined (or community-defined) A, B and C.

This motivates the question:

Question: What are the differences between p-value and Z-score?

And the subquestion:

Question: Are there situations where the p-value and Z-score of the same statistic might suggest opposite hypotheses? Are the first and second conditions listed above essentially the same?

GaBorgulya
  • 3,253
  • 15
  • 19
Douglas S. Stones
  • 6,931
  • 4
  • 16
  • 18

3 Answers3

10

I would say, based on your question, that there is no difference between the three tests. This is in the sense that you can always choose A, B, and C such that the same decision is arrived at regardless of what criterion you are using. Although you need to have the p-value be based on the same statistic (i.e. the Z-score)

To Use the Z-score, both the mean $\mu$ and variance $\sigma^2$ are assumed to be known, and the distribution is assumed normal (or asymptotically/approximately normal). Suppose the p-value criterion is the usual 5%. Then we have:

$$p=Pr(Z>z)<0.05\rightarrow Z>1.645\rightarrow \frac{X-\mu}{\sigma}>1.645\rightarrow X > \mu+1.645\sigma$$

So we have the triple $(0.05, 1.645, \mu+1.645\sigma)$ which all represent the same cut-offs.

Note that the same correspondence will apply to the t-test, although the numbers will be different. The two tails test will also have a similar correspondence, but with different numbers.

probabilityislogic
  • 22,555
  • 4
  • 76
  • 97
8

A $Z$-score describes your deviation from the mean in units of standard deviation. It is not explicit as to whether you accept or reject your null hypothesis.

A $p$-value is the probability that under the null hypothesis we could observe a point that is as extreme as your statistic. This explicitly tells you whether you reject or accept your null hypothesis given a test size $\alpha$.

Consider an example where $X\sim \mathcal{N}(\mu,1)$ and the null hypothesis is $\mu=0$. Then you observe $x_1=5$. Your $Z$-score is 5 (which only tells you how far you deviate from your null hypothesis in terms of $\sigma$) and your $p$-value is 5.733e-7. For 95% confidence, you will have a test size $\alpha=0.05$ and since $p<\alpha$ then you reject the null hypothesis. But for any given statistic, there should be some equivalent $A$ and $B$ such that the tests are the same.

Gary
  • 1,469
  • 1
  • 13
  • 9
  • 3
    @Gary - a p-value doesn't tell you to reject or not any more than a Z-score. They are just numbers. It is only the decision rule which determine accepting or rejecting. This decision rule could equally well be defined in terms of a Z-score (e.g the $2\sigma$ or $3\sigma$ rule) – probabilityislogic Mar 19 '11 at 12:09
  • @probabilityislogic I agree with you. Indeed, you could construct some test based on $Z$ score threshold but it does not allow you to explicitly define a test size in the classical sense (i.e. in terms of probability). This sort of criteria might be trouble some if your distribution has thick tails. When you construct a test, you explicitly define a test size and thus the $p$-value immediately tells you whether you accept or reject, which is the point I was trying to make. – Gary Mar 20 '11 at 08:01
  • @gary - not really, for the p-value makes no reference to alternatives. So it can't be used to directly compare alternatives. For example, take $H_0:\mu=0$ vs $H_A:\mu=-1$. The the p-value for $H_0$ stays the same $5\times 10^{-7}$. So you say "reject the null" which means "accept the alternative" and declare $\mu=-1$. But this is absurd, nobody would do this, but the p-value rule you use here does this. Put another way, the p-value rule you described is not invariant with respect to what is called the "null hypothesis" (resolution coming) – probabilityislogic Mar 20 '11 at 12:50
  • (cont'd) The resolution of the apparent absurdity is note that the p-value is not an "absolute" test, but a relative one, defined with an implicit alternative hypothesis. In this case, the implicit alternative is $H_{imp}:\mu=5$. You can see this by noting that if I calculate the p-value of $H_A$ I get $1\times 10^{-9}$, which is smaller than the p-value for $H_0$. Now in this example, the "implicit alternative" is easy to find by intuition, but it is much harder to find in more complex problems, where nuisance parameters or no sufficient statistic. – probabilityislogic Mar 20 '11 at 12:59
  • @probabilityislogic what you seem to be describing is the power of the test. I've made no suggestions to "accepting the alternative" which is not the same as "rejecting the null." In fact, the implicit alternative is not what you described but rather the two sided test $H_1:\mu\neq0$. The only point I was making is that the $p$-value seems more rigorous and a choice of $Z$ score threshold, which may or may not come from a normal distribution, seems more arbitrary. – Gary Mar 21 '11 at 02:35
  • 1
    @Gary - the p-value is no more rigorous just because it is a probability. It is a monotonic 1-to-1 transformation of the Z-score. any "rigor" that is possessed by the p-value is also possessed by the Z-score. Although if you are using a two sided test then the equivalent is the absolute value of the Z-score. And in order to compare $H_1:\mu\neq 0$ to the null, you have to take a "minimax" approach: which is to pick the sharp hypothesis which is most supported by the data and consistent with $H_1$. Unless you can demonstrate how to calculate $P(X|\mu\neq 1)$ – probabilityislogic Mar 21 '11 at 14:18
6

$p$-value indicates how unlikely the statistic is. $z$-score indicates how far away from the mean it is. There may be a difference between them, depending on the sample size.

For large samples, even small deviations from the mean become unlikely. I.e. the $p$-value may be very small even for a low $z$-score. Conversely, for small samples even large deviations are not unlikely. I.e. a large $z$-score will not necessarily mean a small $p$-value.

SheldonCooper
  • 1,452
  • 9
  • 8
  • if the sample size is large, then the standard deviation will be small, hence the Z-score will be high. I think you may discover this if you tried a numerical example. – probabilityislogic Mar 19 '11 at 12:11
  • 1
    Not really. Suppose you sample from N(0, 1). Then your std will be about 1 regardless of sample size. What will get smaller is the standard error of the mean, not standard deviation. p-values are based on SEM, not on std. – SheldonCooper Mar 19 '11 at 22:19
  • The Z-score is (observed-mean)/(standard deviation). But the mean and standard deviation are of the observed statistic, not of the population from which components of it were drawn. My slack terminology has been caught here. However, if you are testing the mean, then the appropriate standard deviation in the Z-score is the standard error, which gets smaller at the same rate as the p-value. – probabilityislogic Mar 20 '11 at 00:08