4

I'm trying to compare two samples to see if they come from same distribution. I'm not just interested in the measures of central tendency but in the overall shape and nature of the distributions. Based on what I have read on CV and elsewhere, K-S test seems like a good fit for the data I have.

The test, however, is very sensitive to the differences between the two. There are many differences it detects with significant p-values (< 1e-3) that are not really practically significant.(1) The ones I'm really looking for have much lower p-values (< 1e-10), but I'm not sure how to justify using an arbitrarily arrived threshold.

Are there ways to find out a threshold or create a transform function on p-values to have a reasonable cutoff? Any other methods I can use in tandem? Compute an empirical p-value somehow?

Note that the two samples have ties.

Thanks!

Edit:

As it seems from the initial answers, this is problem that requires understanding the data itself so as to be able to distinguish between practically significant versus statistically significant differences. It'd be great if one can explain general strategies to overcome such situation.

Community
  • 1
Vivek Rai
  • 141
  • 5
  • Out of curiosity, how many observations do you have for each sample? – Jonathan Lisic Jun 28 '16 at 20:39
  • @JonathanLisic About 500. – Vivek Rai Jun 28 '16 at 21:34
  • 1
    If you use a wrench to hammer in screws, don't complain that the resulting mess is because something is wrong with the wrench or the screws ... if you care about "practical significance" *don't do an ordinary hypothesis test*, because it simply doesn't deal with that issue. It sounds like you're interested in something like an effect size, so specify what you want to identify and measure that. – Glen_b Jun 29 '16 at 00:05
  • @Glen_b Yes, you are correct. I've only started to think about the problem and that's what I am trying to understand -- _what I want to identity and measure that_. It is in the same process that K-S test came up. Can you talk a bit about general strategies to do what you suggest? Thanks! – Vivek Rai Jun 29 '16 at 06:39
  • You're asking me for a general strategy for figuring out what you want? My general strategy is to ask something like "what are you really trying to achieve?" – Glen_b Jun 29 '16 at 06:44
  • @Glen_b As in, I understand that I want to figure out whether two distributions are same or different. However, I do not have a quantitative way to say that. I can look at the two and say, 'oh yes, they are totally apart' or 'hmm, kinda same!'. I'm just trying to formalize this thought process and that's where the test comes into the play. But, if insufficient, what additional things I should consider to open further line of investigation. Thanks! – Vivek Rai Jun 29 '16 at 06:50
  • Without the whole population, you cannot ever conclude they're the same -- and you have again maneuvered yourself into thinking in hypothesis test terms which you previously acknowledged doesn't solve your problem. You see when you say "kind of the same"? You need to specify that more precisely. It may be that an equivalence test would suit you (which will require you to be quite precise about what equivalence is). These are discussed in a number of posts on site. – Glen_b Jun 29 '16 at 06:54

1 Answers1

4

You have run into one of the many problems with null-hypothesis significance testing. Specifically, the mere falsity of the null hypothesis doesn't say much about a population. A test has told you that two samples aren't from literally identical populations, but that's not what you actually care about. What you actually care about is up to you; it sounds like you care about how different the populations are. The solution is not to try to transform $p$-values ("when all you have is a hammer", etc.) but to use a more appropriate technique. One tack you could take is to use each sample to estimate a population distribution, as with kernel density estimation, then compare your estimates to each other. But it really depends on your specific data and the specific problem you want to solve.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68
  • Thanks! To add additional info about data: Mostly, the population distribution is normal except in few cases where it may have two peaks (that's where I am not able to use mean/median as a good measure of difference). – Vivek Rai Jun 28 '16 at 21:49
  • If you want more specific help, you need to provide much more detail including what problem you're trying to solve, where the data came from, what variables you have available, etc. – Kodiologist Jun 28 '16 at 23:41
  • If you can make parametric assumptions about your data, I'm not sure K-S is the way to go. – HEITZ Jun 29 '16 at 06:54