10

I'm a newcomer in the problem of multiple comparisons. I wonder how to calculate confidence intervals for Holm-Bonferroni method?

I know that for Bonferroni method we can just change the confidence level from $1-\alpha$ to $1-\frac{\alpha}{m}$.

Does this method also work for Holm-Bonferroni?

$\bf{Edit}:$ It seems that H-B method doesn't provide a procedure to correct the conf. interval. But would you comment on can I use one method for p-value correction and the other method for interval correction?

Harvey Motulsky
  • 14,903
  • 11
  • 51
  • 98
Macaronnos
  • 279
  • 2
  • 9
  • Maybe this can help ? https://cran.r-project.org/web/packages/multxpert/multxpert.pdf –  Aug 25 '15 at 16:53

1 Answers1

5

[This answer is completely rewritten from yesterday.]

First nomenclature. The Holm method is also called the Holm step-down method, or the Holm-Ryan method. Those are all the same. Whichever of those names you use, there are two alternative calculations. The original Holm method is based on Bonferroni. An alternative slightly more powerful method is based on Sidak instead, so is called the Holm-Sidak method.

The Holm method can be used for multiple comparisons in a variety of contexts. Its input is a stack of P values. One use is following ANOVA, comparing pairs of means while correcting for multiple corrections. When this is done, as far as I can see, it is very rare to report confidence intervals (corrected for multiple comparisons, so properly called simultaneous confidence intervals) as well as conclusions about statistical significance and multiplicity adjusted P values.

I've found two papers that explain how to compute such confidence intervals, but they differ.

Serlin, R. (1993). Confidence intervals and the scientific method: A case for Holm on the range. Journal of Experimental Education, 61(4), 350–360.

enter image description here

Ludbrook, J. MULTIPLE INFERENCES USING CONFIDENCE INTERVALS. Clinical and Experimental Pharmacology and Physiology (2000) 27, 212–215

enter image description here

For the comparisons with the smallest P values, the two methods are the same (but one uses C as the # of comparisons and the other uses m) . But for the comparisons with larger P values, the two methods differ. For the comparison with the largest P value, Ludbrook would compute the 95% CI normally, with no correction for multiple comparisons. Serlin would use the same adjustment for all comparisons with an adjusted P value greater than 0.05 (assuming you want 95% intervals), so the intervals for the comparisons with large P values would be wider than the ones that Ludbrook method generates.

Both methods use the Bonferroni approach, but could be easily adjusted to the Sidak approach.

Any thoughts on which method is correct/better?

Harvey Motulsky
  • 14,903
  • 11
  • 51
  • 98
  • If you have a P-value then you should be able to get a confidence interval. A one-tailed P-value indicates that the null hypothesis is at the boundary of a 100*(1-P)% confidence interval. Perhaps you could iteratively adjust the null until the P-value comes out as alpha for the confidence interval width desired. – Michael Lew Jun 25 '15 at 02:54
  • But note that multiplicity adjustments, by the nature of the frequentist paradigm, are not prescribed by theory and are somewhat arbitrary. They are not necessarily linked with confidence intervals. There are cases, for example, in group sequential testing where one could stop early rejecting $H_0$ and still have the multiplicity-corrected confidence interval for the treatment effect include zero. – Frank Harrell Jul 25 '15 at 23:38