1

I am trying to work out is there is any association between occurrence or not of an event in around 500 individuals and their randomised binary grouping (intervention vs placebo). However I also want to see if this holds true when broken down by one of 5 locations in which the individuals were treated.

A chi square test on the entire group shows a significant result. However if I repeat the test on each of the 5 subgroups only one subgroup is significant (<0.05) and all of the others are non-signficant.

Further still I have then corrected for multiple testing by multiplying the p-values by 6 (5 tests for each centre and then 1 test for entire group). This does not affect the subgroups but does affect the overall group which is no longer significant after multiple testing correction. I know this issue is controversial. Had I just stuck with the original hypothesis and performed one chi square test I would be saying it is significant but once I do subgroup testing I am then having to say this is not the case. Does anyone have any suggestions on how to approach this problem?

user49166
  • 21
  • 3

2 Answers2

1

The test across the entire group is an omnibus test, and does not need to be adjusted for multiple comparisons. So you would be correcting for $m=5$ multiple comparisons.

Community
  • 1
Alexis
  • 26,219
  • 5
  • 78
  • 131
1

(1) As @Alexis wrote, you do not need to correct for the global hypothesis.

(2) It is easier to detect an effect somewhere than to actually pinpoint where it originates. It is thus not rare than a global (omnibus) test is rejected, but some or even none of the post-hoc tests are not.

(3) With 5 hypotheses, there is not much power to be gained with different multiplicity control schemes (but do try BH for FDR control). It seems that you have 50 independent observation for each effect estimate. You should have a lot of power for any reasonably sized effect. Maybe you should change your test statistic.

JohnRos
  • 5,336
  • 26
  • 56