Can Bonferroni be applied for dependent multiple tests?

Question

Suppose we have 3 classes (A, B and C). Their members took a math test and a philosophy test.

What I want to know is which class performs better on math than the others, and which class performs better on philosophy. They task two different tests, but I want to discuss these subjects completely separately.

If there is no correlation between the scores of math test and those of philosophy test, I think we can simply apply Bonferroni correction. Since we have 3 classes, the significance level is 0.05/3. Then use t-test (or something similar), and say A and B got significantly different scores in the math test but the difference in the philosophy test is not significant, for instance. (Here we used t-test 6 times in total since we have to consider 2 subjects times 3 classes.) Is this idea correct?

What if there is a correlation, meaning students who are good at math tend to be bad at philosophy and vice versa? Do I have to lower the significance level? If I have to, what significance level should be used?

Answer 1

In order to show that the Bonferroni correction controls the familywise error rate you do not need to assume independence, so the type I error will be controlled familywise if you do the Bonferroni correction. That ''proof'' is based on the Boole inequality and that inequality holds in cases of dependence and independence.

So if your significance level is $\alpha$ and you perform $n$ tests, independent or not, and each individual test is done at a significance of $\alpha/n$ then the familywise error for all the tests will be controled at the level $\alpha$, meaning that the probability of a type I error at the family level will be lower than or equal to $\alpha$.

So it may be that the correction is ''conservative'', i.e. the familywise type I error could be stricly lower than $\alpha$, or, so to say, ''you make too few'' type I errors.

At first glance this does not seem like a problem: who would one have a problem with having too few (type I) errors ?

Now there is a trade-off between the power of a test and the probability of a type I error; the lower the probability of a type I error, the higher the probability of a type II error and thus the lower the power of a test.

So if the Bonferroni correction is conservative, it will still control the familywise error at the level $\alpha$ but at a levem strictly lower than $\alpha$. As a too low type I error probability implies a loss of power, it follows that, in cases where the Bonferroni correction is conservative, you loose power !

It can be shown that the Bonferroni correction is conservative when the tests are dependent.

To conclude: you do not need independence for applying Bonferroni, it will still control the familywise error, but in the case of dependence between tests it will be conservative and in that case, even if the familywise type I error is controled, this results in a loss of power.

Note: The Holm procedure controls the type I error familywise in the same way as Bonferroni, the Holm procedure will also be conservative for dependent tests, but less (or equally) conservative as Bonferoni.

Note: The Sidak correction assumes independence.

EDIT 21-09-2016

A. FWER is applicable to your case

FWER is needed whenever you use one and the same sample to test a famliy of hypothesis. This is the case for you e.g. if you want to show that the classes perform different on math, you will have to test three hypothesises, i.e.

1. $H_0^{(1)}: \mu_{Am}=\mu_{Bm}$ versus $H_1^{(1)}: \mu_{Am} \ne \mu_{Bm}$
2. $H_0^{(2)}: \mu_{Am}=\mu_{Cm}$ versus $H_1^{(2)}: \mu_{Am} \ne \mu_{Cm}$
3. $H_0^{(3)}: \mu_{Bm}=\mu_{Cm}$ versus $H_1^{(3)}: \mu_{Bm} \ne \mu_{Cm}$

Where $\mu_{ct}$ is the mean score of class $c$ on test $t$, so e.g. $\mu_{Am}$ is the mean of class $A$ on maths.

There is no doubt that this is a Family of three hypothesises. If you perform each of these three hypothesises at a significance level of $\alpha$ then your type I error will be larger than $\alpha$, so in order to control type I error ''familywise'' you will have to do ''something'' to reduce it to the level of $\alpha$.

Bonferroni is one possibility, but Bonferroni is about controlling the familywise error rate, so there is no doubt that it is applicable to your case. A nice introduction can be found at this link

B. Detailed analysis of your test

In your comment below you are more specific about what you want to do, I cite your comment:

You say: ''What I want to know is, for example, class A does better on math than B does. I will not conclude like class A is better at studying than B is because A got higher scores on at least one subject (either of the subjects or maybe both). Is this right?''

This reminds me at a discussion I had with @amoeba, @Wayne and @Anoldmaninthesea in this post What's wrong with ''multiple testing correction'' compared to ''joint tests''?.

My point is that you must precisely define what you want to test. .

If you want to check whether the three classes perform differently on math, then you should test $H_0: \mu_{Am}=\mu_{Bm}=\mu_{cm}$ versus $H_1: \mu_{Am} \ne \mu_{Bm} \text{ or } \mu_{Am} \ne \mu_{Cm} \text{ or } \mu_{Bm} \ne \mu_{Cm}$. If you know the joint distribution of the class scores on math, then you can do a joint test (see What's wrong with ''multiple testing correction'' compared to ''joint tests''?).

If you do not know that joint distribution then you can do a family of tests, if you control type I error rate !

The family of tests that you can perform is the three tests I mentioned supra. If you want to control FWER at the level $\alpha$ then you should do a correction like e.g. Bonferroni.

However, according to your comment that I cited supra, you don't want to test whether the classes perform differently on math, but whether the classes perform differently in studying better, meaning that they perform better on math OR on philosophy. This implies a different test: $H_0: \mu_{Am}=\mu_{Bm}=\mu_{Cm} \text{ AND }\mu_{Ap}=\mu_{Bp}=\mu_{Cp}$ versus .... (the opposite).

This can be replaced by a family of six hypothesises. If you have a family of six hypothesises then you should divide $\alpha$ by six, as @Björn said in his comment below your question. However, if there is dependence amongst the tests, then dividing by six will lead to a conservative FWER control and to loss of power as I explained supra.

Why is that ? Well I think that is the reason for your question. If you would know the dependence between the results of math and philosophy and assume it is perfect, i.e. if the scores are between 0 and 10, then assume that the points of philosophy are 10-the points of math, so assume that there is perfect dependence.
In that case I know that if the classes score differently on maths then they will also score differently on philosophy (because of the dependence assumed), so I only have to do the test for maths (and this is a family of three tests).

If the dependence is not perfect, then you will have something in the middle.

But what holds is that, even in the case of dependence, the Bonferroni correction controls FWER at the level $\alpha$, however, if there is dependence then this will result in a loss of power (meaning that you will reject too few null hypothesises if you divide by six).

Answer 2

To me this sounds like a 2-Way ANOVA situation (if I understand your question correctly). You have class and subject as factors and score as the outcome or dependent variable. Below is an example of what I would do, incl. an example dataset. The analysis has been done using R but the procedure is the same with any other software. However, note that if you are doing t-tests / ANOVA et al., the variance in your residuals has to be roughly equal (you can and should assess this graphically) as well as normally distributed.

Build example dataset (since it uses the sample() function, values won't be the same):

scores <- data.frame(class=rep(c("A","B","C"), each=12),
                     subject=(rep(c("math","phil"), each=6, 6)),
                     score=sample(c(0:100), size=36, replace=T))

Look at data table (is this what you have in principle?):

> scores
   class subject score
1      A    math    40
2      A    math     8
3      A    math    27
4      A    math    94
5      A    math    31
6      A    math    19
7      A    phil    19
8      A    phil    69
9      A    phil    58
10     A    phil    87
11     A    phil    58
12     A    phil    53
13     B    math    61
14     B    math     1
15     B    math    42
16     B    math    55
17     B    math     8
18     B    math    76
19     B    phil    93
20     B    phil    41
21     B    phil    78
22     B    phil     3
23     B    phil    72
24     B    phil    72
25     C    math    94
26     C    math    98
27     C    math    58
28     C    math    52
29     C    math    78
30     C    math    61
31     C    phil     4
32     C    phil    41
33     C    phil    33
34     C    phil     5
35     C    phil    98
36     C    phil     7
37     A    math    40
38     A    math     8
39     A    math    27
40     A    math    94
41     A    math    31
42     A    math    19
43     A    phil    19
44     A    phil    69
45     A    phil    58
46     A    phil    87
47     A    phil    58
48     A    phil    53
49     B    math    61
50     B    math     1
51     B    math    42
52     B    math    55
53     B    math     8
54     B    math    76
55     B    phil    93
56     B    phil    41
57     B    phil    78
58     B    phil     3
59     B    phil    72
60     B    phil    72
61     C    math    94
62     C    math    98
63     C    math    58
64     C    math    52
65     C    math    78
66     C    math    61
67     C    phil     4
68     C    phil    41
69     C    phil    33
70     C    phil     5
71     C    phil    98
72     C    phil     7

Ran ANOVA for main factors and interaction (however what exactly you want to specify depends on your question) and check output:

fit <- aov(score ~ class + subject + class:subject, data=scores)
> summary(fit)

Df Sum Sq Mean Sq F value  Pr(>F)    
class          2    367     183   0.239 0.78823    
subject        1      8       8   0.010 0.91904    
class:subject  2  15507    7753  10.091 0.00015 ***
  Residuals     66  50712     768                    
---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

In this case, the interaction is significant and neither of the main effects is. You could follow up with multiple mean comparisons (e.g. Tukey's HSD test) for the significant interaction:

> TukeyHSD(fit, "class:subject")

  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = score ~ class + subject + class:subject, data = scores)

$`class:subject`
                    diff         lwr       upr     p adj
B:math-A:math   4.000000 -29.2146663 37.214666 0.9992416
C:math-A:math  37.000000   3.7853337 70.214666 0.0203469
A:phil-A:math  20.833333 -12.3813330 54.048000 0.4472734
B:phil-A:math  23.333333  -9.8813330 56.548000 0.3198057
C:phil-A:math  -5.166667 -38.3813330 28.048000 0.9974039
C:math-B:math  33.000000  -0.2146663 66.214666 0.0524698
A:phil-B:math  16.833333 -16.3813330 50.048000 0.6733007
B:phil-B:math  19.333333 -13.8813330 52.548000 0.5312747
C:phil-B:math  -9.166667 -42.3813330 24.048000 0.9647034
A:phil-C:math -16.166667 -49.3813330 17.048000 0.7096646
B:phil-C:math -13.666667 -46.8813330 19.548000 0.8315925
C:phil-C:math -42.166667 -75.3813330 -8.952000 0.0052198
B:phil-A:phil   2.500000 -30.7146663 35.714666 0.9999243
C:phil-A:phil -26.000000 -59.2146663  7.214666 0.2097783
C:phil-B:phil -28.500000 -61.7146663  4.714666 0.1336128

This tells me that there is a significant difference in math scores between class C and A as well as a significant difference between math and philosophy scores in class C. Now it's getting interesting because you have to speculate why this may be the case :D

There is also plenty of information regarding these analyses here on Cross Validated and you should have a look at them.

Answer 3

@Stefan If you do a single global test, ANOVA needs no adjustment. However, if you look at the p-values for the interaction and the two main effects (or just one of them), you are not doing a single global test, you are doing two or three or so (depending on exactly what you do). Obviously they are somewhat correlated, but they are not a single test.