2

If I have two samples, with one 30-times larger than the other, which nonparametric test statistics should be considered for checking the null of equal distribution?

I am using Kolmogorov-Smirnov test to check that the two samples come from the same distribution. Any mistake in this? Is there any other test statistics to be used?

smillig
  • 2,336
  • 28
  • 31
aravind ramesh
  • 129
  • 6
  • Could you please provide the sample sizes explicitely? Note that you can use [other tools](http://stats.stackexchange.com/questions/28431/what-are-good-data-visualization-techniques-to-compare-distributions/28442#28442) to sustain your conclusions. There are other test that you can also consider [(see)](http://normaldeviate.wordpress.com/2012/07/14/modern-two-sample-tests/). –  Nov 06 '12 at 10:17
  • @Procrastrinator Sample-A 55, Sample-B 1498. – aravind ramesh Nov 06 '12 at 10:24
  • @Procrastinator Yes they are continuous with ties also in the data. For that I am using jitter function and ks.boot function in R to get the most appropriate result. I usually plot boxplot for my analysis. – aravind ramesh Nov 06 '12 at 10:30
  • I have used `nboots=1000` I will check using `10000` and let you know. – aravind ramesh Nov 06 '12 at 10:52
  • @Procratinator No major difference in p-value. `earlier-0.12(@ n=1000) Now-0.11(@ n=10000)` – aravind ramesh Nov 06 '12 at 10:56
  • @Procrastinator Please have a look at [my data](http://www.4shared.com/zip/frq3olek/Data.html) – aravind ramesh Nov 06 '12 at 11:28
  • I have applied several graphical and numerical tools, they look very similar. Particularly, the empirical CDFs are very similar, for this reason the KS test produces a large p-value. I would say that you will hardly find a test that rejects the hypothesis that those samples come from different distributions. Check for example `plot(density(x),type="l",col="red"); points(density(y),type="l",col="blue")` and `plot(ecdf(x)); lines(ecdf(y))`. –  Nov 06 '12 at 11:46
  • You can also conduct a simple permutation test with the code in [this answer](http://stats.stackexchange.com/questions/35819/equivalent-of-kolmogorov-smirnov-test-for-integer-data/35868#35868). The resulting p-value is around $0.5$. –  Nov 06 '12 at 11:58

2 Answers2

2

Jittering your data before testing is problematic (jittering is more used for plots so the points are not on top of each other, but the viewer can still understand that there were ties). With jittering you are comparing datasets that are mixtures of the original and the jittering distribution and adding the jittering could artificially increase or decrease the p-value.

The problem with ties in the KS test is that the p-values are computed based on assumptions that no ties are possible, so the work around is to not use the computed p-values but find your own. One possibility is to do a permutation test: run the KS test but ignore the p-value but record the KS test statistic; now combine the 2 sets of data and randomly choose out of the combined data a new set that is the same size as one of the original (and the rest will represent the other group); run the KS test on these new sets and again just record the test statistic. Repeat this process a bunch of times (2,000 or so) and the recorded test statistics will represent the distribution under the null hypothesis, the proportion of the test statistics that are more extreeme than your original one is the p-value.

You mentioned that you thought the null should be rejected based on a plot of the data, perhaps the methods in this paper:

 Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

would appeal to you instead. The simplified version is that they generate several plots of the data under the assumption of the null being true (permuting similar to above) and show those plots along with the same plot of the original data. If you cannot figure out which is the plot of the original data then that supports the null hypothesis, if the plot of the original is very obvious then that argues against the null. The vis.test function in the TeachingDemos package for R helps implement this test.

If you have already decided that there is a difference and just want a p-value less than 0.05 then you can use SnowsPenultimateNormalityTest that is also in the TeachingDemos package (but be warned, the documentation for that function is considered more useful than the function itself).

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • Aha, just checked out the code of `SnowsPenultimateNormalityTest`. Brilliant!! By the way: did that ever convince anyone? I mean, anyone who is just asking for a p-value lower than 0.05? – gui11aume Jan 06 '13 at 02:23
  • @gui11aume, I think I have convinced a few people to think before doing a normality test, hopefully nobody has actually used my test for a serious study. – Greg Snow Jan 07 '13 at 20:50
1

As fas as I know, there is no particular reason you can't use it with very different sample sizes.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • Thanks for supporting my opinion for K-S test but when I perform this test ,several times I get a p-value of 0.08 to 0.15 which is not considered good for rejecting Null. But by looking the data set we can definitely say that Null should be rejected.That is why I am thinking of using any other test where I can get a p-value less than 0.05. – aravind ramesh Nov 06 '12 at 10:14
  • 2
    You said: "*But by looking the data set we can definitely say that Null should be rejected*" -- assuming you're carrying it out correctly, then you're probably noticing some aspect that the test is less sensitive to than you'd like. That probably suggests a more focused test statistic. ... what causes you to conclude that the null should be rejected? – Glen_b Nov 06 '12 at 11:13
  • On the biological reasoning and some published reports it should be Null should be rejected. Please have a look at [My Data](http://www.4shared.com/zip/frq3olek/Data.html) – aravind ramesh Nov 06 '12 at 11:26
  • 2
    Those might create and *expectation* that it would be rejected, but statistical tests look at the *data*, not biological reasoning or published reports. Rather than get me to download all your data and analyze it (unless you're asking for my consulting services), just draw the two ECDFs... and explain what it is about *the data* that tells you the null ought to be rejected. – Glen_b Nov 06 '12 at 11:37
  • @aravindramesh ... that way, we can try to tease out what it is you regard as a relevant difference, which would allow an improvement in power against the alternatives you see as important. – Glen_b Nov 06 '12 at 22:40