2

I want to test wether population A is significantly different from population B (at 5% confidence interval). I need to do it for each variable using an un-paired t-test, since my data are normally distributed (tested with the Kolmogorov-Smirnov test, kstest function), and that it comes from different conditions. I am using the ttest2 MATLAB function.

Anyway, the t-test rejected the NULL-hypothesis of the mean populations being equal for all the variables. This is surprsing for me, as from the boxplot I expected the populations to be quite similar, except for variable 4 (see boxplot below). I red that the power of the t-test is higher when the samples are many, but decreases if the size of the two samples differ a lot. In my case, POP A has 2724 samples, and POP B has 504728.

In the light of such, one uestion arises: am I performing the right test? Should I say that the samples are significantly different for all the studied variables?

boxplot of my data

umbe1987
  • 195
  • 1
  • 1
  • 8

1 Answers1

1

I don't think it's a problem of unequal sample sizes. It's more about the hypothesis testing nature of a t-test which is based on standard errors. For large samples they can become very narrow as they are in the denominator of the formula $SE=\frac{SD}{\sqrt{N}}$. Unless you are interested in identifying tiny mean differences, a test for effect sizes which are based on standard deviations and are therefore sample size-independent might be more suitable. I think this post would also be of help.

Community
  • 1
StevenP
  • 504
  • 3
  • 13
  • 1
    Thanks for the fast and useful response. Would you suggest any reference to perform Effect Size analysis with MATLAB, and maybe suggest one test I could perform on my data in the light of what I wrote? – umbe1987 Jul 14 '15 at 12:47
  • 1
    For independent samples you could compute the Cohen's D effect sizes. A detailed way of interpreting them is presented [here](http://www.bwgriffin.com/gsu/courses/edur9131/content/EffectSizeBecker.pdf). I am not a MATLAB user, but I found [this](http://uk.mathworks.com/matlabcentral/fileexchange/32398-measures-of-effect-size-toolbox) which I think is what you need. According to the comments Hedges' g is one manifestation of Cohen's D. – StevenP Jul 14 '15 at 13:08