2

I am trying to determine if there is a mean difference between two test groups. The raw data (continuous scores) is distributed according to a long tail distribution.

Each test group has more than 10K observations.

After a lot of reading I concluded that I can probably use the Welch t-test to calculate a 95% confidence interval for the mean difference.

Is this correct or am I violating any important assumptions by doing that?

Cheers, Marcus

Marcus
  • 21
  • 1
  • Do both groups show the same long tail? Do you really need a test of the *mean*, or just that one tends to be higher than the other? – gung - Reinstate Monica Apr 13 '16 at 17:31
  • Yes, both groups' distributions are from the same family of distributions. A confidence interval of mean or median difference would be preferable (so that I can say something about the effect size). – Marcus Apr 14 '16 at 06:00

2 Answers2

1

As with other T-tests, Welch's T-test hinges on the use of the central limit theorem to ensure that the sample means of the two samples are (approximately) normally distributed. This is what gets us to the (approximate) T-distribution for the test statistic. Now, it is not necessary for your samples to be normally distributed, but they must meet the requirements of one of the central limit theorems so that the sample means are normally distributed. (See further discussion on this issue in comments below.)

To meet the requirements of the classical CLT (the Lindberg-Lévy theorem) the two populations must each have finite variance, and this would exclude some long-tailed distributions. Finite variance requires that the tails of the distribution each decrease faster than cubic decay. To check if your distributions have tails that decrease faster than cubic decay (and therefore have finite variance) you should construct a tail plot for each sample, and compare this to a line showing cubic decay; see e.g., this related answer. If you are unable to establish the conditions for the CLT, and it looks like they might not hold, then the requirements of the T-test may not be valid, and the test may give poor results.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • The (Welch-)$t$-test *does* assume normally distributed populations as @Glen_b explains [here](https://stats.stackexchange.com/a/253318/21054). The Wikipedia page you linked to also states this. – COOLSerdash Jun 28 '19 at 05:32
  • I have asserted an *approximate* T-distribution, not an exact T-distribution, so the assertion depends on the degree to which one regards closeness in distribution as "approximate" equality. As Glen points out in the linked answer, even without an underlying normal population, for large $n$ it is possible to establish asymptotic normality of the sample mean, and an asymptotic chi-squared distribution for the sample variance. These are asymptotically uncorrelated so long as the distribution has finite skew, which is enough to get what I regard as a close approximation to the T-distribution. – Ben Jun 28 '19 at 05:54
  • (It is also possible to get a closer approximation by adjusting the degrees of freedom to account for the kurtosis of the underlying distribution --- see e.g., [O'Neill 2014](https://www.tandfonline.com/doi/full/10.1080/00031305.2014.966589).) – Ben Jun 28 '19 at 05:56
  • Thanks. I was under the impression that the mean and variance are only independent in a normal distribution (e.g. [Lukacs 1942](https://projecteuclid.org/download/pdf_1/euclid.aoms/1177731647))? – COOLSerdash Jun 28 '19 at 06:01
  • 1
    They are only independent in that case, but their covariance in the general case is $\mathbb{C}(\bar{X}, S^2) = \gamma \sigma^2 / n$, which approaches zero as $n \rightarrow \infty$ (so long as $\gamma < \infty$). Now, obviously being asymptotically uncorrelated is weaker than independence, but it is a decent first-order approximation for large $n$. – Ben Jun 28 '19 at 06:10
  • 1
    Very interesting, thanks. – COOLSerdash Jun 28 '19 at 06:12
  • 1
    Correction to comment above - that should be $\mathbb{C}(\bar{X},S^2) = \gamma \sigma^3/n$ (i.e., the sigma should be cubed, not squared). – Ben Jun 28 '19 at 06:21
  • I find it interesting that the skewness plays an important role in this. In an other [post](https://stats.stackexchange.com/a/30053/21054), it was shown that the (asymptotic) performance of the $t$-test is much more sensitive to deviations from normality in the form of skewness than kurtosis. – COOLSerdash Jun 28 '19 at 06:31
  • 1
    Yes, that sounds about right - I would think that deviations in lower-order moments would be more of a problem than deviations in higher-order moments. – Ben Jun 28 '19 at 06:46
-1

This video shows a clear explanation and simple explanation of comparing two means. Three assumptions are made for this tool:

  1. Homogeneity of variance

  2. Normal distribution of populations (according to you, this is your case)

  3. Each value is sampled independently

If you can assume those three:yes, you can use Welch's t-test or student's t-test

galtor
  • 123
  • 2
  • 6
  • It is the second assumption that I am not sure I understand: "Normal distribution of populations (according to you, this is your case)". Raw data is definitively not normally distributed - it follows a long tail distribution (the same family of distributions for both groups). – Marcus Apr 14 '16 at 06:02
  • Is it the raw data or the sample test statistic that needs to be normally distributed? (my groups are large and after running simulations I can see that my test statistic, i.e. the sampled mean difference, is indeed normally distributed) – Marcus Apr 14 '16 at 06:10