0

I want to look at whether there is a significant difference in mean scores on a personality questionnaire between a high autism trait sample (HAQ) and a low autism trait sample (LAQ).

However, the sample sizes of the two groups are incredibly different with 134 people in the LAQ and 10 people in the HAQ.

Furthermore, the LAQ sample is normally distributed, but the HAQ isn't (most likely because it's a small sample).

The Levene's test seems to indicate that the variances are equal in the two groups.

I was wondering what I should do to account for the differences in normality and differences in sample size? Should I bootstrap the t-test? Should I use a Mann-Whitney U test?

Any references to sources would be greatly appreciated.

Thanks!

Babette
  • 1
  • 2
    This is classic case for permutation testing, see e.g. https://stats.stackexchange.com/questions/295246/how-to-apply-bootstrapping-in-this-case/295764#295764 – Tim Mar 07 '18 at 17:55

2 Answers2

2

A few comments to begin:

First, the sample size has nothing to do with whether the underlying distribution is actually Normal; it has much to do with whether you can detect deviations from normality (or whatever the assumed distribution is.) Having said that, though, I would not be at all surprised to find the HAQ sample was quite non-normal, so let's continue with that.

Second, Levene's test will have low power with one of the groups having such a small sample size (as will pretty much every other test for that matter.) This means that it is unlikely to flag any differences in variance that aren't really large. So... I wouldn't rely much on assumptions about equality of variances.

On to the question! Given your situation, I would avoid the t-test and probably use the Mann-Whitney U test (see When to use the Wilcoxon rank-sum test instead of the unpaired t-test? for a discussion of the equivalent Wilcoxon rank-sum test), but @Tim's comment about permutation testing and his excellent answer in the link he provided is a good one too. To a large extent, it comes down to exactly what you want to test. If the distributions are symmetric - which the HAQ distribution might well not be - testing for differences between means is essentially the same as testing whether one distribution's values are greater than the other, which is what Mann-Whitney tests for, otherwise Mann-Whitney doesn't actually test for differences between means. See @ttnphns's answer to the above-linked question for more on this. If you really are interested in the means themselves, though, then constructing a permutation test would be more on-target.

Note also that with your sample sizes you probably won't be able to construct a true permutation test, in which you enumerate all permutations of assignment to HAQ and LAQ groups, but will have to bootstrap instead, which brings us back to your suggestion of bootstrapping the t-test. There is a right way and a wrong way to do this if your objective is to test equality of means: see Using bootstrap under H0 to perform a test for the difference of two means: replacement within the groups or within the pooled sample for an explanation (the accepted answer is where the issue is addressed.)

jbowman
  • 31,550
  • 8
  • 54
  • 107
1

Imbalanced samples do not fundamentally affect analysis or inference. However, the power of a study depends strongly on the sample size of the smallest group to be compared. Below I discuss ways that, aside from distributional considerations, may strongly improve the generalizability and precision of the analysis. Following these recommendations, any testing procedure you feel is (or isn't) appropriate can be considered. With N=10, the concern that normal approximations are not being met especially with irregular data, is justified. I suspect the variability of the HAQ sample demonstrates some bimodal and skewed properties. That may simply be an aspect of those people: it doesn't imply the mean is not a useful summary, so I might hazard against rank tests.

In observational studies of this form, there are often contributors to the outcomes that are worth bearing into mind. For example, the profundity of autism is strongly affected by age, gender, educational setting, household structure, and so on. With such a small sample, it may sound crazy to exclude participants: but recall the excess of LAQ participants does not necessarily improve power.

I suggest sorting participants to the extent possible based on these factors and obtaining a matched subsample at a 1:1 or 1:2 ratio as you can. You may in fact see there is a subdistribution of LAQ that aligns more closely with HAQ, and thus any form distributional comparison (t-test, Wilcoxon, or permutation test) has greater generalizability and precision.

AdamO
  • 52,330
  • 5
  • 104
  • 209