P-value in a two-tail test with asymmetric null distribution

Question

My situation is as follows: I want, through a Monte-Carlo study, to compare $p$-values of two different tests for statistical significance of an estimated parameter (null is "no effect - parameter is zero", and the implied alternative is "parameter is not zero"). Test A is the standard "independent two-sample t-test for equality of means", with equal variances under the null.

Test B I have constructed myself. Here, the null distribution used is an asymmetric generic discrete distribution. But I have found the following comment in Rohatgi & Saleh (2001, 2nd ed, p. 462)

"If the distribution is not symmetric, the $p$-value is not well defined in the two-sided case, although many authors recommend doubling the one-sided $p$-value".

The authors do not discuss this further, nor do they comment on the "many authors suggestion" to double the one-sided $p$-value. (This creates the question "double the $p$-value of which side? And why this side and not the other?)

I was not able to find any other comment, opinion or result on this whole matter. I understand that with an asymmetric distribution although we can consider an interval symmetric around the null hypothesis as regards the value of the parameter, we will not have the second usual symmetry, that of probability mass allocation. But I do not understand why this makes the $p$-value "not-well defined". Personally, by using an interval symmetric around the null hypothesis for the values of the estimator I see no definitional problem in saying "the probability that the null distribution will produce values equal to the boundaries of, or outside this interval is XX". The fact that the probability mass on the one side will be different than the probability mass on the other side, does not appear to cause troubles, at least for my purposes. But it is rather more probable than not that Rohatgi & Saleh know something that I don't.

So this is my question: In what sense the $p$-value is (or can be) "not well defined" in the case of a two-sided test when the null distribution is not symmetric?

A perhaps important note: I approach the matter more in a Fisherian spirit, I am not trying to obtain a strict decision rule in the Neyman-Pearson sense. I leave it up to the user of the test to use the $p$-value information alongside any other information to make inferences.

Answer 1

If we look at the 2x2 exact test, and take that to be our approach, what's "more extreme" might be directly measured by 'lower likelihood'. (Agresti[1] mentions a number of approaches by various authors to computing two tailed p-values just for this case of the 2x2 Fisher exact test, of which this approach is one of the three specifically discussed as 'most popular'.)

For a continuous (unimodal) distribution, you just find the point in the other tail with the same density as your sample value, and everything with equal or lower likelihood in the the other tail is counted in your computation of p-value.

For discrete distributions which are monotonically nonincreasing in the tails, it's just about as simple. You just count everything with equal or lower likelihood than your sample, which given the assumptions I added (to make the term "tails" fit with the idea), gives a way to work it out.

If you're familiar with HPD intervals (and again, we're dealing with unimodality), it's basically like taking everything outside an open HPD interval that's bounded in one tail by your sample statistic.

[To reiterate -- this is likelihood under the null we're equating here.]

So at least in the unimodal case, it seems simple enough to emulate Fisher's exact test and still talk about the two tails.

However, you may not have intended to invoke the spirit of Fisher's exact test in quite this way.

So thinking outside that idea of what makes something 'as, or more extreme' for a moment, let's head just slightly more toward the Neyman-Pearson end of things. It can help (before you test!) to set about defining a rejection region for a test conducted at some generic level $\alpha$ (I don't mean you have to literally compute one, just how you would compute one). As soon as you do, the way to compute two tailed p-values for your case should become obvious.

This approach can be valuable even if one is conducting a test outside the usual likelihood ratio test. For some applications, it can be tricky to figure out how to compute p-values in asymmetric permutation tests... but it often becomes substantially simpler if you think about a rejection rule first.

With F-tests of variance, I've noticed that the "double one tail p-value" can give quite different p-values to what I see as the right approach. [I insist, for example, that it shouldn't matter which group you call "sample 1", or whether you put the larger or the smaller variance in the numerator - yet with some common approaches these apparently reasonable conditions are violated.]

[1]: Agresti, A. (1992),
A Survey of Exact Inference for Contingency Tables
Statistical Science, Vol. 7, No. 1. (Feb.), pp. 131-153.

Answer 2

A p-value's well-defined once you create a test statistic that partitions the sample space & orders the partitions according to your notions of increasing discrepancy with the null hypothesis. (Or, equivalently, once you create a set of nested rejection regions of decreasing size.) So what R. & S. are getting at is that if you consider either high or low values of a statistic $S$ to be interestingly discrepant with your null hypothesis you still have a little work to do to get a proper test statistic $T$ from it. When $S$ has a symmetric distribution around nought they seem to leap to $T=|S|$ without much thought, & therefore regard the asymmetric case as presenting a puzzle.

Doubling the lowest one-tailed p-value can be seen as a multiple-comparisons correction for carrying out two one-tailed tests. After all, following a two-tailed test, we're usually very much inclined to regard any doubt cast on the truth of the null as favouring another hypothesis whose direction is determined by the observed data. A proper test statistic is then $t=\min(\Pr_{H_0}(S<s),\Pr_{H_0}(S>s))$, & when $S$ has a continuous distribution the p-value is given by $2t$.^†

When $S$ has a continuous distribution, the approach to forming a two-tailed test shown by @Glen_b—defining the density of $S$ as the test statistic: $T=f_S(S)$—will of course produce valid p-values; but I'm not sure that it was ever recommended by Fisher, or that it's currently recommended by neo-Fisherians. If at first glance it appears more principled somehow than doubling the one-tailed p-value, note that having to deal with probability density rather than mass means that that the two-tailed p-value thus calculated may change when the test-statistic is transformed by an order-preserving function. For example, if to test the null that a Gaussian mean is equal to nought, you take a single observation $X$ & obtain $1.66$, the value with equal density at the other tail is $-1.66$, & the p-value therefore $$p=\Pr(X > 1.66) +\Pr(X<-1.66)=0.048457+0.048457=0.09691.$$ But if you consider it as testing the null that a log-Gaussian geometric mean is equal to one & take a single observation $Y$ & obtain $\mathrm{e}^{1.66}=5.2593$, the value with equal density at the other tail is $0.025732$($=\mathrm{e}^{-3.66}$), & the p-value therefore $$p=\Pr(Y>5.2593) +\Pr(Y<0.025732)=0.048457+0.00012611=0.04858.$$

Note that cumulative distribution functions are invariant to order-preserving transformations, so in the example above doubling the lowest p-value gives \begin{align}p=2t&=2\min(\Pr(X<1.66),\Pr(X>1.66))\\&=2\min(\Pr(Y<5.2593),\Pr(Y>5.2593))\\&=2\min(0.048457,0.951543)\\&=2\times 0.048457=0.09691.\end{align}

A kind of sequel to this answer, discussing some principles of test construction in which the alternative hypothesis is explicitly stated, can be found here.

† When $S$ has a discrete distribution, writing

$$p_\mathrm{L} = \Pr_{H_0}(S\leq s)$$ $$p_\mathrm{U} = \Pr_{H_0}(S\geq s)$$

for the lower & upper one-tailed p-values, the two-tailed p-value is given by

$$ \Pr(T\leq t) = \begin{cases} p_\mathrm{L} + \Pr_{H_0}(P_\mathrm{U} \leq p_\mathrm{L}) & \text{when}\ p_\mathrm{L} \leq p_\mathrm{U}\\ p_\mathrm{U} + \Pr_{H_0}(P_\mathrm{L} \leq p_\mathrm{U}) & \text{otherwise} \end{cases} $$

; i.e. by adding to the smaller one-tailed p-value the largest achievable p-value in the other tail that does not exceed it. Note that $2t$ is still an upper bound.