2

A standard problem in statistics would be something like the following.

$$ H_0: \theta = \theta_0 $$

$$ H_a: \theta \ne \theta_0 $$

This is a two-sided test.

What if we did the following two tests?

$$ _1H_0: \theta = \theta_0 $$

$$ _1H_a: \theta > \theta_0 $$

$$ _2H_0: \theta = \theta_0 $$

$$ _2H_a: \theta < \theta_0 $$

The complaint that I see is that this raises $\alpha$, since there are multiple testing issues at play, and then if we consider multiple tests, this becomes equivalent to the two-sided test.

There is more than one way to adjust a p-value, and that approach, to me, implies a simple Bonferroni correction.

There are better adjustments than Bonferroni.

Could we keep $\alpha$ at the level we want (say $0.05$ or $0.01$) but increase power by using a better correction than Bonferroni?

Dave
  • 28,473
  • 4
  • 52
  • 104
  • 3
    Your one-sided nulls should have greater/smaller or equal rather than just equal. It’s true that you calculate the p-value at the boundary of that interval, but the null is technically a compound one. The hypotheses need to partition the full sample space. – dimitriy Jul 19 '21 at 15:58
  • Aside: your nomenclature with respect to null and alternative hypotheses is a bit inconsistent, since (a) it implies that the alternative hypotheses are not logical complements of their respective null hypotheses, and (b) leaves undefined possibilities in the sample sample space a la for $_{1}_{}:\theta= \theta_0$ and $_{1}_{}:\theta>\theta_0$ the condition where $\theta – Alexis Jul 19 '21 at 17:38

2 Answers2

1

One practical issue is that the two alternative sub-hypotheses in this context are are mutually exclusive and the experimenter is really just interested in rejecting one of them. So for example, suppose the Holm procedure is applied instead of the Bonferroni, then you either reject the test with the lowest p-value if this is less than $\alpha/2$ or reject none at all, which is exactly the same situation if you used Bonferroni since it is not possible to reject both hypotheses anyway.

You can try other adjustments but each would have its own added assumptions and requirements, some of which are not applicable to this setup where there is high negative dependence between the two hypotheses. For example, Sidak's procedure may fail to control family-wise error rate under negative dependence.

variancekills
  • 43
  • 5
  • Negative dependence appears [*very* difficult to provide an example of](https://stats.stackexchange.com/questions/111756/). – Alexis Jul 19 '21 at 17:45
  • Alzheimers study. Endpoint 1 = time to complete cognitive exam. Endpoint 2 = number of errors. Because the endpoints are negatively correlated, so are the test statistics and the one-sided $p$-values. – BigBendRegion Jul 19 '21 at 19:32
  • From my understanding, the context of the question is dealing with a single hypothesis test and wanting to shift from testing non-directionally, to testing each direction. So in that case the p-values of each direction would be negatively related to each other, right? – variancekills Jul 19 '21 at 20:34
0

Yes. Assuming you have more than one such two-sided test, you may consider:

The Holm-Šidak adjustment gives much more powerful sequential rejection control over the family-wise error rate under dependency.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(65–70), 1979.

The Benjamini-Hochberg adjustment gives even more powerful control of the false discovery rate.

Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300.

Alexis
  • 26,219
  • 5
  • 78
  • 131