Repeated Measures ANOVA and the Bonferroni post hoc test different results of significantly

Question

I am doing an Repeated Measures ANOVA and the Bonferroni post hoc test for my data using R project. The ANOVA gives a significantly difference between the data but not the Bonferroni post hoc test.

> anova(aov2)
            numDF denDF   F-value p-value
(Intercept)     1  1366 110.51125  <.0001
time            5  1366   9.84684  <.0001

while

> pairwise.t.test(x=table.metric2$value, g=table.metric2$time, p.adj="bonf")

        Pairwise comparisons using t tests with pooled SD 

data:  table.metric2$value and table.metric2$time 

    Su1 Su2 Su3 Su4 Su5
Su2 1   -   -   -   -  
Su3 1   1   -   -   -  
Su4 1   1   1   -   -  
Su5 1   1   1   1   -  
Su6 1   1   1   1   1  

P value adjustment method: bonferroni

These are my data with the code used

plot <- c(rep(1:275))

Su1 <- c(13.5584,0.0000,2.0710,0.4826,1.2761,1.6690,3.5188, 13.7578,0.0000,0.0000,0.0004,0.0000,0.0000,0.0000,4.4634,3.0151,2.1719,5.2861,4.9651,0.7908,0.0000,0.0000,0.0000,0.0000,
0.0000,5.2749,5.4706,4.4416,3.2166,0.0000,0.1929,0.0000,0.0000,0.0000,0.0000,0.0000,4.6765,1.7761,4.3579,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,6.1794,2.4194,1.4319,0.0000,0.0327,0.0000,0.4633,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0018,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.5425,0.5274,2.0883,0.6024,0.0000,0.0018,0.0000,0.0000,0.0015,0.0012,0.0000,0.0000,0.0000,
0.4560,2.1256,0.0295,0.0328,0.0000,1.6447,0.0428,0.0067,0.0058,0.0000,0.0000,0.0000,0.0001,0.3317,0.2898,3.5134,0.1539,0.0199,0.0000,0.0494,0.1159,2.0976,0.0644,0.9730,
0.0010,0.5074,0.0003,0.0000,0.1188,1.8818,0.0000,0.1213,0.3585,7.3932,0.5492,0.0045,0.9879,0.0010,0.7625,0.1695,0.1211,0.3164,2.6750,3.8926,0.0000,3.4626,0.0000,5.8339,
6.7315,0.0244,4.8770,2.6237,2.3700,0.5338,0.0215,3.2196,1.9811,3.3825,3.3929,1.5426,0.9165, 10.6561,3.2154,4.1531,5.3381,3.9432,4.8675,0.0047,0.0026,0.2058,1.8509,0.3697,
0.3131,0.0707,4.7908,6.4087, 10.3670,5.7662,4.0460,3.2571,9.1767,0.0116,0.0908,0.0053,0.1480,0.9063,5.4331,5.7945, 14.4097,6.9635,7.0637,0.1064,9.9095, 11.8432, 10.0234,0.0000,
0.0491,5.0472,5.3094,5.1657, 14.3944,7.6244,0.0034,1.4953, 14.7658,6.1775,7.1567,0.0296,0.0911,3.5552,4.9543,3.1200,1.9774,0.0000,4.1663,0.0000,2.3672,0.0638,1.8952,4.1948,
6.4229, 10.7573,0.0008,1.3818,6.0011,3.6791,9.7816,1.5203,0.8616,1.5483,5.4174,2.7070,2.1627,0.0000,1.7360,3.7292,2.4638,7.4498,4.2343,6.8263,3.2410,0.0001,0.0001,9.7424,
4.2861,2.9912,0.4316, 11.6082,2.0138,0.0002,1.8783,0.9934,0.2983,1.4013,0.1429)

Su2 <- c(10.4361,0.0000,2.3346,0.5769,1.3392,1.5908,3.5759, 13.3183,0.0005,0.0000,0.0019,0.0000,0.0000,0.0000,4.4862,3.0418,2.3991,5.4263,4.9456,0.6907,0.0000,0.0007,0.0000,0.0000,
0.0000,5.1065,5.1748,4.2888,3.3633,0.0000,0.1791,0.0000,0.0000,0.0000,0.0000,0.0000,5.3485,1.9216,4.2880,0.0000,0.0001,0.0001,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,6.3664,2.2888,1.4636,0.0000,0.0282,0.0000,0.5838,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0013,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0002,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.6240,0.6401,1.9324,0.6263,0.0000,0.0020,0.0000,0.0001,0.0039,0.0018,0.0000,0.0000,0.0000,
0.4321,1.9815,0.0528,0.0350,0.0000,1.6519,0.0328,0.0153,0.0171,0.0000,0.0000,0.0000,0.0000,0.3966,0.2957,3.4653,0.1473,0.0038,0.0000,0.0774,0.1180,2.2780,0.0611,0.9490,
0.0024,0.8337,0.0000,0.0000,0.1531,1.9778,0.0000,0.1171,0.3485,7.2378,0.7476,0.0028,1.2072,0.0015,0.7425,0.1352,0.1908,0.3092,2.3735,3.9768,0.0000,3.8786,0.0000,6.5420,
6.8490,0.0256,5.8268,2.7856,2.4640,0.6399,0.0215,5.3411,1.7939,3.3401,3.2942,1.4990,0.9264, 10.6864,3.3749,4.7978,5.2929,3.8639,5.3890,0.0027,0.0486,0.2105,1.8514,0.3526,
0.3146,0.0950,4.8061,6.2244, 10.1131,5.8538,3.8861,4.4240,9.5952,0.0255,0.0533,0.0085,0.1742,1.0188,7.7153,5.4663,14.6060,6.8725,6.7284,0.0841, 10.2016, 11.3384,9.8938,0.0000,
0.0749,5.0774,7.2876,5.2040, 14.7609,7.7862,0.0005,1.4099, 14.8639,6.8801,7.2587,0.0125,0.0513,3.1338,7.1539,3.1733,1.9729,0.0000,4.5081,0.0000,2.1572,0.0452,1.8866,8.0198,
9.7868, 10.8746,0.0000,1.5011,6.0825,3.6705,9.9171,1.7091,0.8267,1.4186,6.4235,2.9303,1.9019,0.0000,1.5869,4.4028,2.4186,7.7739,4.6728,8.7722,3.9859,0.0001,0.0001, 10.1583,
5.4758,2.8977,0.9716, 11.5342,2.6111,0.0000,1.7497,1.0041,0.3273,1.1953,0.2289)

Su3 <- c(12.5808,0.0000,2.7244,0.1119,1.2633,0.9702,2.4513,2.4419,0.0014,0.0000,0.0000,0.0000,0.0019,0.0105,3.5392,2.4033,2.1847,5.5912,4.8628,0.9449,0.0000,0.0000,0.0000,0.0000,
0.0000,4.2928,5.0754,3.4644,2.8663,0.0066,0.0043,0.0000,0.0000,0.0000,0.0000,0.0000,3.5292,1.4846,4.6333,0.0000,0.0003,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,7.0110,2.7508,2.0824,0.0001,0.0001,0.0000,0.0770,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0023,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.4546,0.1491,2.2391,0.4464,0.0000,0.0007,0.0000,0.0000,0.0000,0.0238,0.0000,0.0000,0.0000,
0.5663,1.5734,0.0247,0.0259,0.0000,0.6823,0.0101,0.0000,0.1171,0.0000,0.0000,0.0000,0.0000,0.1110,0.0960,2.2888,0.1338,0.0000,0.0000,0.0294,0.1273,2.3082,0.0081,0.6343,
0.0021,0.1690,0.0000,0.0000,0.3456,1.7644,0.0000,0.0175,0.1027,9.8583,0.5133,0.0002,0.8704,0.0000,0.0850,0.1183,0.0949,0.0369,2.8741,1.0065,0.0000,4.6330,0.0000,7.7170,
7.7201,0.0038,3.7342,2.6457,1.6262,0.2836,0.0008,2.2052,2.6186,3.1951,5.4121,0.5787,1.4276, 12.5830,5.0891,3.7475,4.8317,0.3612,5.0634,0.0001,0.0002,0.2396,2.8372,0.5667,
0.0024,0.0396,4.4980,6.6203,11.8869,7.5642,4.4144,1.1202,7.8870,0.0214,0.0323,0.0000,0.1492,1.0881,3.4938,6.4025, 13.7996,5.3475,7.2365,0.4289, 10.5106, 12.1750,7.8348,0.0000,
0.0104,0.0126,4.7145,3.5840,11.1274,7.1017,0.4464,1.3384, 14.9060,4.1969,7.7008,0.0214,0.0022,2.6731,1.7724,3.0937,1.3026,1.1184,1.0817,0.0000,1.6675,0.0879,1.6432,4.3482,
3.5927,3.3646,0.0000,1.2088,2.6632,2.6712, 10.0635,6.1788,0.9486,0.8011,2.5326,5.1218,0.1574,0.0000,1.7523,5.2613,2.8762,6.7293,7.9969,3.7011,1.9242,0.0002,0.0000,6.8359,
3.4735,4.8440,1.0125,4.3772,1.9227,0.0034,1.0078,2.7654,0.0246,0.4001,0.2570)

Su4 <-  c(NA,0.0001,NA,0.3616,0.2848,1.5804,3.2575,6.9722,0.0000,0.0053,0.0001,0.0000,0.0002,0.0000,4.9185,3.6967,1.9675,7.7624,4.0516,0.7658,0.0001,0.0018,0.0000,0.0000,
0.0000,5.4964,5.2645,5.0756,1.9665,0.0011,0.3481,0.0001,0.0000,0.0000,0.0000,0.0000,4.6501,1.6375,1.1133,0.0016,0.0063,0.0007,0.0000,0.0001,0.0008,0.0000,0.0000,0.0000,
0.0000,5.9308,2.0694,0.3611,0.0035,0.0126,0.0000,0.4887,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0001,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0002,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.6246,0.4996,1.2273,0.5902,0.0000,0.0016,0.0000,0.0000,0.0004,0.0010,0.0000,0.0000,0.0000,
0.4431,1.6631,0.0234,0.0230,0.0000,1.4833,0.0134,0.0321,0.0283,0.0000,0.0000,0.0000,0.0000,0.2446,0.1740,3.4582,0.0757,0.0092,0.0000,0.0202,0.1099,2.1932,0.0565,0.9499,
0.0176,0.2146,0.0001,0.0000,0.1009,1.8096,0.0000,0.0413,0.2970,7.6015,0.4349,0.0121,0.8901,0.0024,0.4629,0.0843,0.0934,0.2790,2.5034,3.0369,0.0000,4.1673,0.0000,6.2014,
5.6363,0.0211,5.2860,2.7722,1.7316,0.3452,0.0099,2.6113,1.8183,3.2228,3.1208,1.4761,0.9326,9.7211,4.4282,4.0727,5.1978,3.3293,5.8666,0.0013,0.0413,0.0958,1.3760,0.2844,
0.1587,0.0197,3.4947,6.2988, 10.5278,5.6255,3.7482,4.0839,9.5720,0.0113,0.0060,0.0005,0.2592,1.1092,6.5303,5.7979, 15.0240,6.6722,7.0578,0.0507,9.6608, 12.2310,7.6133,0.0002,
0.0147,4.6729,4.1950,5.4444, 13.3876,8.7654,0.0001,1.3864, 15.5325,5.2633,9.7407,0.0160,0.0069,3.3050,7.6701,2.8048,1.1736,0.0000,4.8370,0.0000,1.4801,0.0014,1.3862,7.8413,
3.9453,11.0588,0.0000,1.7270,6.0988,4.2754, 10.7927,1.1768,0.6381,0.1779,6.2290,2.9297,0.4520,0.0012,0.9822,6.6041,2.8147,6.9679,3.5493,9.5997,4.5469,0.0007,0.0004,6.7922,
5.2609,2.4192,0.0459, 10.4127,3.9877,0.0000,1.1876,0.8595,0.5107,0.8049,0.1749)

Su5 <- c(10.4824,0.0000,2.0018,0.2817,1.0086,1.1640,2.6718, 11.2732,0.0000,0.0000,0.0003,0.0000,0.0000,0.0000,5.1846,3.1089,1.6050,5.3045,5.1410,0.7458,0.0000,0.0000,0.0000,0.0000,
0.0000,5.8211,5.6094,4.2910,2.6412,0.0000,0.0404,0.0000,0.0000,0.0000,0.0000,0.0000,4.8222,1.7778,1.6938,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,5.3706,2.1161,0.5845,0.0000,0.0154,0.0000,0.6280,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0001,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.5967,0.6631,1.2776,0.8574,0.0000,0.0010,0.0000,0.0000,0.0016,0.0010,0.0000,0.0000,0.0000,
0.4013,1.7534,0.0341,0.0360,0.0000,1.8029,0.0163,0.0280,0.0541,0.0000,0.0000,0.0000,0.0000,0.2873,0.2288,3.3035,0.0943,0.0277,0.0000,0.0273,0.1230,1.1606,0.0553,0.9261,
0.0006,0.2793,0.0001,0.0000,0.0702,1.7861,0.0000,0.0441,0.6919,7.2771,0.4737,0.0001,0.9996,0.0028,0.7776,0.0760,0.2064,0.3322,2.1804,3.8545,0.0000,3.8377,0.0000,6.0567,
6.7315,0.0168,6.6603,2.2872,2.7505,0.6174,0.0235,2.5142,1.3832,3.0431,2.3944,1.4855,0.5978,9.6032,3.2907,3.7126,5.1397,3.7415,5.0454,0.0003,0.0361,0.0761,1.4128,0.3045,
0.4881,0.0392,3.8213,5.8874,9.9054,4.9779,3.2351,4.7589,9.2551,0.0108,0.0184,0.0086,0.1905,1.0657,8.1882,5.5578, 14.6536,7.3801,7.8668,0.0315,9.0087, 10.1677,9.6868,0.0001,
0.0797,5.6032,5.4432,5.9958, 15.4570,8.1095,0.0000,1.1854, 14.0602,5.9429,8.4364,0.0056,0.0163,4.0671,5.5083,3.2078,1.8711,0.0000,6.0628,0.0000,1.8110,0.0941,1.8252,8.3102,
9.3829,5.5418,0.0000,1.4030,8.6327,4.3226,9.3724,1.6725,1.0198,1.4480,6.7463,1.7035,1.3542,0.0000,1.4079,4.2148,2.6589,7.3225,4.6582,9.1156,4.0312,0.0000,0.0000,9.2176,
5.5112,2.1917,0.5516,9.7061,2.7556,0.0000,1.0919,1.0696,0.5361,1.0160,0.1019)

Su6 <- c(9.3294,0.0001,1.7365,0.2030,0.5796,0.3794,2.4745,0.0104,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,2.2045,2.8210,1.0607,4.9375,0.0892,1.1530,0.0000,0.0000,0.0000,0.0000,
0.0000,5.5562,5.9813,4.6396,3.0815,0.0000,0.1245,0.0000,0.0000,0.0000,0.0000,0.0000,4.4952,1.3822,5.3511,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,2.4023,0.3491,2.0235,0.0000,0.0469,0.0000,0.3316,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,
0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.0000,0.5361,0.5083,2.4144,1.6541,0.0000,0.0009,0.0000,0.0000,0.0007,0.0084,0.0000,0.0000,0.0000,
0.7155,2.4862,2.8153,2.0548,0.0000,1.0062,0.1725,0.0028,0.0404,0.0000,0.0000,0.0000,0.0000,2.0952,1.8754,2.3209,0.1329,0.0025,0.0000,0.0134,0.1656,1.9201,0.0624,0.9148,
0.2393,1.8717,0.0000,0.0000,0.2733,1.3219,0.0000,0.0502,0.4294,6.5172,0.4368,0.0001,2.5930,0.0024,1.1465,0.0857,0.1336,0.1791,2.2427,1.7424,0.0000,4.5865,0.0000,7.4015,
5.1587,0.0953,1.9529,1.7191,2.6812,1.0850,0.0133,3.5750,1.3194,2.2280,2.7217,2.5155,0.6812, 10.1853,0.3850,4.3999,3.9028,1.5311,5.2335,0.0006,0.0359,0.2206,1.5833,0.1611,
0.2389,0.2579,4.7568,5.7652, 11.4255,3.1828,1.8413,3.6434,6.6662,0.0148,0.0166,0.0284,0.4646,1.2736,5.3859,6.2033, 14.0715,5.1563,7.2547,0.0163,9.8463,9.7350,7.7685,0.0015,
0.0694,2.4957,5.9204,3.0981, 13.9429,6.0630,0.0000,0.4167, 16.6025,8.0124,6.8630,0.0314,0.0453,5.1017,7.6560,2.2824,1.6168,0.0000,1.5369,0.0000,1.4277,0.1390,1.3599,5.5345,
7.0305,4.4969,0.0000,1.4448,3.2237,3.5196, 11.8150,1.1668,0.5838,1.5561,2.9927,1.8511,1.4603,0.0000,1.2046,1.1346,2.0005,7.2672,6.1411,3.1801,1.8131,0.0005,0.0000,4.1790,
3.8307,1.0645,NA,NA,0.0119,0.0001,1.1278,0.1273,0.0837,0.0863,0.6916)

table.metric <- data.frame(plot,Su1, Su2, Su3, Su4, Su5, Su6)
plot <- table.metric$plot
    table.metric$plot <- NULL
table.metric2 <- stack(table.metric)
plot.metrics = factor(rep(plot,6))
table.metric2[3] = plot.metrics
names(table.metric2) <- c("value", "time", "plot") 
aov2 <- lme(fixed= value~time, random=~1|plot, na.action=na.omit, data=table.metric2)
summary(aov2)
anova(aov2)
pairwise.t.test(x=table.metric2$value, g=table.metric2$time, p.adj="bonf")

For the record, I disagree that this is a duplicate; while the two should not be expected to be the same, in this case, my opinion is that the true issue is that the t-tests are ignoring the repeated measures in the data. — Aaron left Stack Overflow, Jun 24 '15 at 13:39

score 2 · Accepted Answer · edited Apr 13 '17 at 12:44

2

Your t-tests are ignoring the repeated measures in the data. Use paired=TRUE. And just use the Bonferroni-Holm default adjustment; there should be no reason to use regular Bonferroni.

pairwise.t.test(x=table.metric2$value, g=table.metric2$time, paired=TRUE)
##  
##         Pairwise comparisons using paired t tests 
##  
## data:  table.metric2$value and table.metric2$time 
##  
##     Su1    Su2    Su3    Su4    Su5   
## Su2 0.0124 -      -      -      -     
## Su3 0.0241 0.0032 -      -      -     
## Su4 0.9299 0.0115 0.0953 -      -     
## Su5 0.9299 0.0087 0.0937 0.9299 -     
## Su6 0.0038 5e-05  0.9299 0.0186 0.0074
##  
## P value adjustment method: holm

But also see Peter Flom's excellent answer to a related question. These tests get at different things so will not always "agree." Better to decide which test is most appropriate for your question.

As mentioned in the comments, the mixed model assumes equal variance, but the pairwise paired t-tests do not. To get pairwise tests based on the model, which would keep this assumption, you can use the multcomp library, like this (output not shown).

library(multcomp)
summary(glht(aov2, linfct=mcp(time="Tukey")))

In this case, however, they give different results (if you define different as on different sides of the nominal 0.05 cutoff); this is because the variance of the differences between each pair are not equal, the standard deviations have a range of about 3.

ss <- summary(glht(aov2, linfct=mcp(time="Tukey")))
foo <- pairwise.t.test(x=table.metric2$value, g=table.metric2$time, 
                       paired=TRUE)
foo <- as.data.frame(as.table(foo$p.value))
foo <- foo[!is.na(foo$Freq),]
names(foo)[3] <- "pval.pairedt"
foo$pval.pairedt <- format.pval(foo$pval.pairedt, digits=3)
foo$pval.mixed <- format.pval(ss$test$pvalues, digits=3)
foo$Var1 <- factor(foo$Var1, levels=levels(table.metric2$time))
foo$Var2 <- factor(foo$Var2, levels=levels(table.metric2$time))
foo$diff.sd <- sapply(1:nrow(foo), function(i) {
    y1 <- table.metric2$value[table.metric2$time==foo$Var1[i]]
    y2 <- table.metric2$value[table.metric2$time==foo$Var2[i]]
    sd(y1-y2, na.rm=TRUE)
})
print(foo, row.names=FALSE)
## Var1 Var2 pval.pairedt pval.mixed   diff.sd
##  Su2  Su1      0.01239   0.727461 0.4963881
##  Su3  Su1      0.02409   0.008360 1.3264442
##  Su4  Su1      0.92986   0.962717 0.7934766
##  Su5  Su1      0.92986   0.999487 0.7164677
##  Su6  Su1      0.00383   0.000114 1.3635100
##  Su3  Su2      0.00316   2.21e-05 1.4763516
##  Su4  Su2      0.01150   0.227215 0.7914265
##  Su5  Su2      0.00867   0.512340 0.5840229
##  Su6  Su2     4.97e-05   4.83e-08 1.3861764
##  Su4  Su3      0.09527   0.102065 1.3110117
##  Su5  Su3      0.09366   0.024984 1.4534286
##  Su6  Su3      0.92986   0.896211 1.2320563
##  Su5  Su4      0.92986   0.996275 0.7751331
##  Su6  Su4      0.01860   0.003817 1.3068406
##  Su6  Su5      0.00743   0.000494 1.3654174

edited Apr 13 '17 at 12:44

Community

1

answered Jun 24 '15 at 02:37

Aaron left Stack Overflow

6,808
29
51

Your suggesting is (a) ignoring the pooled variance implied by the rANOVA's null hypothesis, and (b) is ignoring the substantive point that with the Bonferroni procedure the meaning of $\alpha$ in the rANOVA no longer obtains, so different results are quite likely (even using the appropriate *post hoc* paired *t* tests with pooled variance). – Alexis Jun 24 '15 at 04:52
(a) True. But with 275 individuals, we have plenty to estimate the variance separately for each pair. And if the variance is different (which should be checked by diagnostics before trusting the mixed model, as well) this is perhaps a better choice anyway. (b) Not sure what you mean by "obtains." Also, are you referring to the use of Bonferroni-Holm or to Peter Flom's answer? – Aaron left Stack Overflow Jun 24 '15 at 13:47
(a) is not a matter of power, it is a matter of understanding what your null hypothesis is, and what that means for your test statistics. (b) Under the the Bonferroni adjustment, $\alpha$ is no longer the probability of observing a single false positive in a single hypothesis test (i.e. where the single null hypothesis is true), whereas that is *precisely* what $\alpha$ means with the rANOVA; therefore, the discrepancy between rANOVA (rejected) and pairwise tests (none, or fewer rejected) is not at all surprising... it is rather expected. – Alexis Jun 24 '15 at 14:11
(a) But the variance assumption is not part of the null hypothesis. (b) I think you're saying that the overall hypothesis will have more power than the pairwise tests because of the pairwise tests correct for multiple comparison. This is not true, at least in the case of a standard ANOVA vs pairwise correction with Tukey's HSD [reference needed, but don't have time at present.] – Aaron left Stack Overflow Jun 24 '15 at 15:56
rANOVA assumes equal within subjects variation, therefore the best estimator of the variance for the *poc hoc* *t* tests is the pooled (within subjects) estimate from the rANOVA (sphericity assumption). – Alexis Jun 24 '15 at 16:02
What I said about (b) was *explicitly* about the meaning of $\alpha$ (i.e. the probability of making exactly one Type I error assuming a single null hypothesis is true). Because the *post hoc* tests entail *more than one* null hypothesis, the $\alpha$ from the rANOVA *cannot* mean the same thing as the $\alpha$ in the *post hoc* tests corrected using the Bonferroni adjustment, therefore it should be no surprise that rejection probabilities differ (whether or not true or false rejections). – Alexis Jun 24 '15 at 16:02
I'm not so sure. The null hypothesis for the main term is $mu_1=mu_2=...mu_k$, and so $\alpha$ is the probability of rejecting the idea that they're all equal. The null hypothesis for each of the pairwise tests is $mu_i=mu_j$, and after the multiple correction, $\alpha$ is the probability of rejecting any of those, and so again rejecting the idea that they're all equal. – Aaron left Stack Overflow Jun 24 '15 at 18:00
And what are the assumptions that attend that null hypothesis, Aaron? One of them is that the within subject variances are equal (without this assumption the *F* test statistic would be more or less uninterpretable). The *t* test statistics also incorporate variance estimates in the denominator… the null of the *post hoc* estimates are *specifically* that they are drawn from the same population as that assumed under the rANOVA's null hypothesis (this differentiates them from ordinary paired *t* tests). If they are from that population, then the best estimate of this variance is the pooled one. – Alexis Jun 24 '15 at 18:50
And again: you don't understand the meaning of $\alpha$. my downvote stands. – Alexis Jun 24 '15 at 18:50
Ouch... I'm out. – Aaron left Stack Overflow Jun 24 '15 at 19:00

Repeated Measures ANOVA and the Bonferroni post hoc test different results of significantly

1 Answers1