0

This is an extension from my last post. It seems to be too long to discuss there.

  • When testing some null $H$ versus alternative $K$ by a test statistic $U(X)$, the p-value for $U$ on a sample $X$ can be defined as the infinimum of size $\alpha$ of the test rule, with varying critical value $c$ while still rejecting the null on $X$: $$p_U(X) := \sup_{F \in H} P(U(Y) \geq U(X) | Y \sim F), \quad i.e. \quad \alpha(U(X)),$$ where $\alpha(c)$ means the size of the test rule $(U, c)$.

  • When the permutation test with the set $G$ of permutations can be applied to the above test (the null, $U$ and $X$), the p-value of the permutation test is said to be $$ p_P(X) := \frac{\# \{\pi \in G: U(\pi X) \geq U(X)\}}{|G|} \\ \stackrel{?}{=} P(U(Y) \geq U(X) | Y \sim F_0) $$ where $F_0 \in H$ is a common distribution of $X$ and $\pi X$ under null, $\pi X$ means applying the permutation $\pi$ to the sample $X = \{x_1, ..., x_n\}$, and the part after $\stackrel{?}{=} $ is what I thought of myself.

Questions:

  1. I was wondering if $p_U(X) = p_P(X)$? I.e., is the p-value of the permutation test equal to the p-value based on $U(X)$, for sample $X$?

  2. If yes, how are they equal when the null is composite?

    • $p_U(X)$ considers all the null distributions $F$'s one by one and then take $\sup_{F\in H}$,

    • $p_P(X)$ seems to equal $P(U(Y) \geq U(X) | Y \sim F_0)$ for a single null distribution $F_0$ of $X$, and doesn't consider other null distributions of $X$.

Thanks!

Community
  • 1
Tim
  • 1
  • 29
  • 102
  • 189
  • 3
    A meta-comment here as on many, many of your questions is that you seem to be reacting to one (unstated) source. So (1) please give a reference (2) note that anything tied to one specific reference is hard to answer unless it is accessible; even with a mass of definitions, the context can be important (3) try reading something else if one source is obscure to you. – Nick Cox Jul 10 '13 at 15:13
  • @NickCox: I try to give references that I can. The definition of the p-value for test statistic $U$ is standard, and can be found in Bickel and Docsum's Mathematical Statistics Vol1. The definition of the permutation p-value is also standard, and can be found from p5 of [a note](http://www.biostat.jhsph.edu/~iruczins/teaching/140.615/notes/n.pnp.pdf), although the part after $\stackrel{?}{=}$ is what I thought of myself. – Tim Jul 10 '13 at 15:24
  • Please finish your edits first. – whuber Jul 10 '13 at 15:45
  • @NickCox: About (2), my questions are about the basic concepts of permutation test and p-value only, and there is no special context attaching to it.(3) I have tried to search in alternative sources that I can. – Tim Jul 10 '13 at 15:47
  • 1
    @Tim, as I mentioned in your earlier post, I contest the claim that that definition you have been relying on is "standard". There's much about it that's quite odd. – Glen_b Jul 11 '13 at 01:03
  • @Glen_b: Can you point out what is odd? – Tim Jul 11 '13 at 01:19
  • @Glen_b: In the equation I gave for p-value of $U$, there is $\sup$ over the null. In the equation for the permutation p-value, there is no $\sup$ over the null. Does it mean that the two p-values are not the same, with the former no less than the latter? – Tim Jul 11 '13 at 01:30
  • "*Can you point out what is odd?*" -- I pointed to an example of the way in which it was odd in the other thread some time ago. – Glen_b Jul 11 '13 at 02:35

0 Answers0