5

Recently learned that Levene's test is a one-way equal-variance ANOVA done on absolute values of residuals calculated from the mean within each group.

For one-way ANOVA the minimal sample size seems to be having at least one group with more than 1 observation. But in the case of Levene's test it gets a bit tricky for me.

For example, if all groups have 2 observations each, the within-group variance will be 0. So the requirement would seem to be at least 1 group with at least 3 observations.

However what about situations where one group has only one sample? I did a few simulations in R using car::leveneTest() and it seems like p-values are not distributed uniformly in the case of 2 groups where one group has only one sample. Here is a demonstration:

library(car)
groups <- factor(c(rep("A", 999), "B"))
ps <- replicate(100, leveneTest(rnorm(1000), groups)[1,3])

> range(ps)
[1] 0.1681269 0.2107370

Basically, after simulating 100 scenarios where group 1 has 999 observations and group 2 has 1 observation, the p-values range from 0.16 to 0.22. The levenTest() function didn't complain, but that might be an oversight in the implementation.

Question: what are the minimum sample size requirements for Levene's test to be valid?

My current take: 2 samples per group with at least one group having 3 but I might have missed something.

Karolis Koncevičius
  • 4,282
  • 7
  • 30
  • 47
  • I think the issue is a question of power. We can do a t-test with 2 points, would we trust it though? Small sample sizes might inflate the Type II errors (*saying there is nothing, when there is something*). – usεr11852 Apr 01 '18 at 15:48
  • @usεr11852 thank you for the first comment. I agree with your note that in practical application this would be a power Issue. However what I am trying to do is implement Levene's test as a function. My worry is about validity - at what sample sizes the function should stop the user from going further. – Karolis Koncevičius Apr 01 '18 at 16:16
  • 1
    Right... Sorry I missed that part! In that sense I would say just two is enough... I would rethink that Levene test as: `anova(lm(absresid ~ myfactor, mydata))` in which case even just two data points (one for each group) would be adequate. Sure we would have 0 DFs and no concept of an $F$ statistic but you know, it is "valid". I would probably give a warning for the DFs but aside that we are golden. (+1 BTW, fun question) – usεr11852 Apr 01 '18 at 16:42
  • Glad to hear the question is entertaining :) I get what you are saying - but is it possible to list all of the situations that should produce a warning somehow? I.E. even if all groups have 2 observations each - after taking `absresid` all of groups will have 1 unique point. As in: `abs((3,7) - (5,5)) = (2,2)`. So based on that I feel cases for warning in ANOVA and in Leven's test should be different. – Karolis Koncevičius Apr 01 '18 at 17:21
  • While amicable, I think you cannot save users from being unreasonable. Warnings on 0-th DFs and/or spreads should be adequate. – usεr11852 Apr 01 '18 at 19:12

1 Answers1

2

As with most hypothesis tests small sample sizes inflate the occurrence of Type II errors, the test is "underpowered". That does not mean though that the test is moot but rather than it has a higher probability of being misleading. Ultimately, Levene's test is an $F$-test and should be treated as one.

I think it will be relevant to give warnings for potentially:

  1. having single observation groups ( $0$ degrees of freedom for the residuals (this will more or less equate to testing that we have more than one observation per group)) and
  2. $0$ within-group variances (no point treating a constant as an R.V.)

Given these two conditions are not met, the findings are "valid" in the sense they are sensible on face value. Note that these are warnings that stem from the "user's question" rather than the "R code's validity". In that sense we do not need to check for a minimum sample size but rather for cases that the sample used is inadequate to provide even an approximate answer. The statistical power of a test is not only a function of the sample size but also of the effect size, so strictly focusing on sample size misses part of the "power" problem.

Probing this a bit further the R code within car::leveneTest actually does an ANOVA on an lm object (Exempt from leveneTest.default: table <- anova(lm(resp ~ group))[, c(1, 4, 5)]) which brings us back to the case that standard ANOVA/lm warnings should probably adequate. In that sense:

A <- data.frame(y = runif(4), g = c(rep("a",2), rep("b",2)) )
car::leveneTest(y ~ g, A)

is a "valid" call and the problem/warning becomes that the lm has an $R^2$ = 1 showing that something went very fishy.

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • Appreciate the answer. Could you also comment on the scenario in the question when one group has only one sample? It doesn't fall under 0 DF or 0 within-group scatter. Yet the p-values seem to not be uniformly distributed under the null. – Karolis Koncevičius Apr 02 '18 at 17:00
  • Maybe I misinterpreter your comment but to me, if we have one sample in one group, it is natural that group will have 0 (on NA depending on your definition) scatter. – usεr11852 Apr 02 '18 at 17:59
  • Sorry if it was not clear enough. Here is a second attempt: you said Levene's test and F-test should be treated the same. But I am stuck on this situation: when group A has 999 samples and group B has only 1 sample - ANOVA returns correct distribution of p-values, and for the Levene's test this is not the case (short simulation in question). So it seems to me that there might be additional restrictions for Levene's test. – Karolis Koncevičius Apr 02 '18 at 18:17
  • Let's run with that scenario... What would the variance of group B? (Let me note that ANOVA itself is not perfect, the fact that it returns some results does not make it a reasonable result) – usεr11852 Apr 02 '18 at 18:18
  • Well, what I am getting at is this: in that scenario - within-group variance is not 0, between group variance is not 0, treatment DF is not 0 and residual DF is not 0. So based on the given answer it should suffice. Yet it doesn't seem to be working. – Karolis Koncevičius Apr 02 '18 at 18:37
  • And it's not a power issue, it just never works. For example here is the test implemented in `car` library: `leveneTest(c(rnorm(999, sd=1000000), 1) ~ rep(c("A","B"), c(999,1)))`. I am testing 999 samples from normal distribution with variance of a million against one sample. The p-value is never below 0.05 even after hundreds of trials. With ANOVA this works as expected - bigger differences produce smaller p-values and we get 5% false positive rate. – Karolis Koncevičius Apr 02 '18 at 18:39
  • With only one subject in a group, the within-group variance has to be either 0 or NA. `A = data.frame( x = c(rnorm(999, sd=1000000), 1),g = rep(c("A","B"), c(999,1) )); aggregate(A$x, by = list(A$g), var)` both ANOVA and Levene are irrational here. There is no second "group", group B is a constant. That's why I am saying check the variances. – usεr11852 Apr 02 '18 at 18:45
  • But within-group variance is averaged from all the groups? So it's not really zero. It's just zero for one group, which is not a problem for an F-test as far as I am aware. – Karolis Koncevičius Apr 02 '18 at 18:46
  • 1
    Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/75421/discussion-between-usr11852-and-karolis-koncevicius). – usεr11852 Apr 02 '18 at 18:46