1

I have two surveys out right now, both with approximately 200K responses. After looking at their descriptive statistics, one of them seem to be getting slightly higher ratings (type int; range(0-5)).

I'd like to evaluate whether this is statistically significant. Reading this post, I saw that the Kolmogorov-Smirnov test is used for similar use cases.

Would it make sense to use the same for my problem?

madsthaks
  • 277
  • 3
  • 11

1 Answers1

7

No, you can't.

The Kolmogorov-Smirnov test is used to test whether two distributions are equal. What you are interested in is whether two distributions have the same mean.

For instance, two distributions $(1,2,3)$ and $(2,2,2)$ are unequal, and the KS test would detect this (except for the sample size, which is too small), but their means are equal, so the KS test is misleading for you.

What you want is a t test, which tests equality of means. It comes in various flavors. You want the "unpaired" one, because your data are unpaired, and you should take a look at your data to decide whether you can assume equal variances in your two populations.

Note, however, that you have a very large sample. Any test will return a statistically (!) significant difference in means even if the difference is tiny, simply because the sample size is so large.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thanks for your response! This is very helpful. So, would you agree that after a certain number of samples (high N), just looking at which mean is higher would be sufficient when deciding which survey to use? – madsthaks Sep 18 '19 at 19:50
  • 1
    Yes, exactly. If all you are interested in is statistical significance. For *substantive* significance, it would make sense to look at how much the means differ. – Stephan Kolassa Sep 18 '19 at 19:51
  • 2
    "...whether you can assume equal variances in your two samples." Do you want "population" there @StephanKolassa ? – Matthew Drury Sep 19 '19 at 04:21
  • 2
    @MatthewDrury: it's always nice to see someone reading my posts *closely* ;-) Yes, of course, I meant "population". Will edit. – Stephan Kolassa Sep 19 '19 at 07:10