3

My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it acceptable for me to run 6 Kruskal-Wallis tests but use a Bonferroni correction ($p'=p/6$). If this is acceptable, is it then reasonable to use 6 subsequent and corrected Dunn's tests to see which variables are significant?

It seems to me that this way of doing it is clunky, but I am not aware of a better way to compare my data.

P.S. I am doing this analysis in MATLAB with the statistics tool box.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
Joe
  • 33
  • 1
  • 3
  • 1
    In what sense are your data not normal? How much data do you have? Are the variables understood as related (eg, measuring different aspects of the same construct)? – gung - Reinstate Monica Jan 14 '15 at 20:19
  • The variables are independent in the sense that one observation can be described by these 6 variables. It's a bit hard to explain what I mean by non-normal but briefly, I used a mahalanobis distance to find observations that were outliers. I now re-grouped my data into 2 groups of these outliers and a much larger group consisting of the rest of the data. That is why now, I want to see how significantly different they are from the rest of the population(and from each other) – Joe Jan 14 '15 at 20:29
  • That information should go in your question – Glen_b Mar 11 '15 at 22:43
  • You might want to consider the answers to this question http://stats.stackexchange.com/questions/139311/using-multiple-comparisons-corrections-in-fisher-p-value-framework/139685#139685 for general guidance on whether it is always sensible to correct for multiplicity of comparisons at all. – Michael Lew Mar 11 '15 at 23:41

1 Answers1

8

You would use the Bonferroni for a one-way test. But let's be clear:

You would not use the Bonferroni adjustment on the Kruskal-Wallis test itself. The Kruskal-Wallis test is an omnibus test, controlling for an overall false-positive rate.

You would use the Bonferroni for post hoc Dunn's pairwise tests. Indeed, Dunn introduced the "Bonferroni" adjustment.

You could also consider using more powerful family-wise error rate adjustment methods, such as the Holm-Sidak method, or still more-powerful false discovery rate adjustment methods, such as the Benjamini-Yekutieli adjustment. These and other multiple comparisons adjustments are implemented specifically for Dunn's test in Stata (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and in R (see http://cran.r-project.org/web/packages/dunn.test/).

You might also consider using the more powerful Conover-Iman post hoc (only) test statistic. This test is implemented for Stata (within Stata type net describe conovertest, from(https://alexisdinno.com/stata)), and for R in the conover.test package, and includes the same selection of multiple comparisons adjustment procedures.

EDIT: I missed that you seem to be wanting something like a multivariate (multiple dependent variables) nonparametric one-way ANOVA. Katz and McSweeney offer such a generalization of the Kruskal-Wallis test, although I am not aware of an implementation in software. They also provide post hoc Scheffé-like univariate and multivariate procedures, which ought to be amenable to multiple comparisons adjustments. Their test is likewise omnibus for all variables.

References
Conover, W. J. (1999). Practical Nonparametric Statistics. Wiley, Hoboken, NJ, 3rd edition.

Conover, W. J. and Iman, R. L. (1979). On multiple-comparisons procedures. Technical Report. LA-7677-MS, Los Alamos Scientific Laboratory.

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.

Katz, B. M. and Mcsweeney, M. (1980). A multivariate Kruskal-Wallis test with post hoc procedures. Multivariate Behavioral Research, 15:281–297.

Alexis
  • 26,219
  • 5
  • 78
  • 131
  • Thanks, I really appreciate your quick and thorough response. What I am trying to understand however is that I get how the kruskal-wallis test is omnibus for one of my 6 variables. But If I am interested in saying that the groups are different in any of the 6 variables wouldn't running multiple KW tests increase the probability of a FP in one of the variables? – Joe Jan 14 '15 at 20:20
  • @Joe It is an omnibus test **for all** of your variables... just like ANOVA is an omnibus test **for all** variables. You perform a single KW test on all vars. Also: you are welcome. – Alexis Jan 14 '15 at 20:30
  • I don't understand. The Kruskal-Wallis test is the non-parametric analog to an ANOVA. It does not handle multiple variables(at least not in MATLAB) – Joe Jan 14 '15 at 20:34
  • By "multiple variables", I think he means multiple Y variables. – gung - Reinstate Monica Jan 14 '15 at 20:36
  • @gung that is correct. – Joe Jan 14 '15 at 20:47
  • @Joe Sorry... I meant "multiple groups." Did you mean multiple variables? Hah... yes... of course you did... hmmm... Revising answer – Alexis Jan 14 '15 at 20:55
  • @gung That'd be "she" – Alexis Jan 14 '15 at 20:59
  • @Alexis, by "he", I was referring to *Joe* (who I just assumed was male based on the name). I'm aware that you are female. I apologize for any confusion or offense. – gung - Reinstate Monica Jan 14 '15 at 21:02
  • @gung None taken. My misunderstanding... I seem to be surfeit of misunderstanding today. :) – Alexis Jan 14 '15 at 21:03
  • 1
    @Alexis, yes. Let me be explicitly clear, I want to find the differences between the three groups in my data. I understand that a KW followed by Dunn's post hoc test I can determine if the groups are different for one variable. But since I am also comparing 5 other variables in the same way it appears as though I would need to correct for that – Joe Jan 14 '15 at 21:04
  • 2
    @Alexis, that's good. My impression is that female answerers are a minority on CV (I have no idea about askers). & I sometimes wonder if there might be something about CV that is unfriendly towards women's participation. As you may know, there is some discussion about unfriendliness / discrimination towards women in online forums. – gung - Reinstate Monica Jan 14 '15 at 21:16
  • @Joe hi there! What method did you go for in the end? I'm facing a similar problem and would just like to know what your solution was. – hbot Aug 24 '15 at 14:17
  • @Joe Hey sorry for bringing up old thread! I'm wondering if you ended up doing a multiple comparison correction on top of Kruskal-Wallis p-values? I am working on same problem and having same question! – Darren Christopher Mar 26 '21 at 04:36
  • @DarrenChristopher My R packages `dunn.test` and `conover.test` and Stata packages `dunntest` and `conovertest` have many multiple comparisons options built in (I would recommend Holm if you need a FWER method, and Benjamini-Hochberg is you need an FDR method; I lean towards the latter). Or, are you asking about multil Kruskal-Wallis tests? – Alexis Mar 26 '21 at 17:22