1

Imagine a 3 by 1 experiment. One group has 1,000 observations, one with 5,000 observations, and 4,000 observations in the last group. I'm trying to see whether the manipulation between groups in the experiment had an effect. Do I need to do anything about the fact that the groups are different sizes, for example by giving greater weight to the group with only 1,000 observations?

Imagine my model is something like.

lm(DV ~ 3by1Variable, data = df).

Where the three by one variable is a factor for whether an observation was in group 1, 2, or 3.

Eric Tim
  • 65
  • 6

1 Answers1

1

I would recommend oneway.test in R, which does not assume the groups have the same variance. The allowance for unequal population variance as reflected in sample variances is much the same as in a Welch t test; the residual degrees of freedom are smaller if sample variances differ. The assumption that the three groups be from normal populations remains.

Fictitious data along lines you describe are sampled in R as follows. Of course, your data will be different. If your data should present issues not covered here, please ask.

set.seed(507)
x1 = rnorm(1000, 50, 7)
x2 = rnorm(5000, 52, 8)
x3 = rnorm(4000, 54, 9)

Here we know that the data were sampled from normal populations. But for your data it would be prudent to look at normal probability plots of the three samples to see if their plots are substantially linear. (As here, larger samples tend to give more-nearly linear plots than smaller ones; minor departures from linearity in the tails are nor worrisome.)

par(mfrow=c(1,3))
 qqnorm(x1, col="red"); qqline(x1)
 qqnorm(x2, col="green2"); qqline(x2)
 qqnorm(x3, col="skyblue2"); qqline(x3)
par(mfrow=c(1,1))

enter image description here

Boxplots give another chance to see whether the samples are roughly symmetrical. However, with normal datasets of this size, multiple moderate 'outliers' are unsurprising.

x = c(x1, x2, x3)
g = c(rep(1,1000), rep(2,5000), rep(3,4000))
boxplot(x~g, col=c("red", "green2", "skyblue2"))
abline(h = 50)

enter image description here

The `oneway test gives a P-value very nearly $0,$ so we reject the null hypothesis that all population mean are equal. [In an ordinary ANOVA, assuming equal population variances, the denominator DF would be nearly 10,000, but is considerably lower here on account of the different sample variances.]

        One-way analysis of means 
     (not assuming equal variances)

data:  x and g
F = 77.195, num df = 2.0, denom df = 2936.9, p-value < 2.2e-16

Ad hoc two-sample Welch t tests indicate that all three means differ. Because of the tiny P-values involved here, it seems that any reasonable method of protecting against 'false discovery' from multiple tests on the same data would still find highly significant differences.

t.test(x1,x2)$p.val
[1] 8.844251e-08
t.test(x2,x3)$p.val
[1] 1.287326e-20

Plots of empirical CDFs (ECDFs) of the three samples show that Group 3 ECDF (blue) plots to the right and below the others, and hence tends to have higher values overall. Similar comparisons can be made between other groups.

enter image description here

plot(ecdf(x3), col="skyblue2", main="ECDFs of Groups")
 lines(ecdf(x1), col="red")
 lines(ecdf(x2), col="green2")
BruceET
  • 47,896
  • 2
  • 28
  • 76