13

I am testing equality of means using Welch's t-test. The underlying distribution is far from normal (more skewed than the example in a related discussion here). I can obtain more data but would like some principled way of determining to what extent to do so.

  1. Is there a good heuristic for making the assessment that the sample distribution is acceptable? Which deviations from normality are most concerning?
  2. Are there other approaches--e.g. relying on a bootstrap confidence interval for the sample statistic--which would make more sense?
Community
  • 1
cohoz
  • 618
  • 5
  • 16
  • 2
    This is a great question. Aside from [Is normality testing "essentially useless"?](http://stats.stackexchange.com/q/2492/22228) (already linked), two more related questions are [How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples?](http://stats.stackexchange.com/q/121852/22228) and [T-test for non normal when N>50?](http://stats.stackexchange.com/q/9573/22228) A good answer to this question would potentially be valuable to readers of these two related questions. – Silverfish Sep 27 '15 at 23:44
  • As far as I know there aren't any principled ways of determining how much data you need for the distribution to be "normal enough." This is because "normal enough" is hard to define, and would depend on how non-normal the underlying distribution is, in addition to the particular way in which you're departing from normality. If you have seriously non-normal data I'd just use a non-parametric test instead. The downside is you wouldn't be able to get confidence intervals which are more useful than lone hypothesis tests. – dsaxton Nov 27 '15 at 15:47
  • 2
    I agree that "normal enough" is hard to define, but every practitioner must make the assessment before reasoning about empirical data, which is why I am surprised how little discussion I have been able to uncover (perhaps I am looking in the wrong places). For the use cases I have in mind here (which feel common enough) non-parametric tests are unsatisfactory compared with collecting more data to ensure a "normal enough" sampling distribution. – cohoz Nov 28 '15 at 04:57

1 Answers1

1

As the t test assume normality, and your underlying distributions are not normal, there cannot be a principled way of determining that the sample distribution is acceptable. However, as the sample size gets "large", the Central Limit Theorem kicks in, and you can use a large sample z-test, which will essentially give you the same answer as a t-test because the t approaches the normal distribution with large samples.

Stats books/courses often imply that at a sample size of 25 or 30 CLT comes into play in a useful way. However, my experience has been that even with sample sizes in the hundreds large sample z-tests can still be pretty poor (e.g., with count data).

In my opinion, a permutation test is a good fit to your problem. It should have equal or better power than canned nonparametric tests (e.g, Mann-Whitney) and you don't have to worry about the normality issue. And, they are fun to write.

Tim
  • 3,255
  • 14
  • 24