2

I have a program in which I need to perform an action based on what results of a Mann Whitney U test are. I know the criteria that need to be met, but I would still like to understand what the results of this test mean.

Scipy's implementation of the test return a value from 0 to .5, and if I understand correctly, .5 means that there is a 100% chance that the two datasets came from the same source. Is this correct?

Glen_b
  • 257,508
  • 32
  • 553
  • 939
Alex
  • 21
  • 1
  • 2
  • 1
    It seems that there has has recently been an overhaul of the Scipy Mann Whitney U test. The [pull request](https://github.com/scipy/scipy/pull/4933) was merged in 2021 so it may be worth updating the package - `pip install --upgrade scipy` - if you like the changes. There are also some much needed changes to [the documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html), which really helped me to understand what is actually going on. Although the reply from Glen_b helped more... – Callum Jun 25 '21 at 15:53

1 Answers1

4

Scipy's implementation of the test return a value from 0 to .5,

If that's actually true, then it's not giving you a p-value.

And indeed, I've just gone and looked. It isn't giving a proper p-value -- though it says it is. It is instead doing something almost guaranteed to make some people do their tests wrong:

The reported p-value is for a one-sided hypothesis, to get the two-sided p-value multiply the returned p-value by 2.

Which one sided test is it doing? Well, if you get the two-sided p-value by doubling it, it's choosing the direction of the test based on the data. Which means you can neither do one-sided nor two sided tests using that number. It's not actually a p-value for any specific test*, but it claims it is.

* not for a two-sided alternative, nor a one-sided greater than alternative, nor a one-sided less than alternative. So it's not a p-value for any of the possible Mann-Whitney tests.

[This is, frankly, appalling. IMO that's really bad software design; it looks almost guaranteed to lead novice users to make mistakes.]

and if I understand correctly, .5 means that there is a 100% chance that the two datasets came from the same source. Is this correct?

Well, let's double it so we are clear we have a two-tailed p-value of 1.0

No, that's not what it would mean at all. A p-value is the probability of a test statistic at least as unusual as the one observed if the null hypothesis were true.

If it's high, it doesn't mean the null is exactly true. It just means you won't see a statistic more consistent with the null -- but that can be produced by situations where the null is false.

You may like to search our site for questions on interpreting p-values.

e.g. these ones may help:

Interpretation of p-value in hypothesis testing

is this interpretation of the p-value legit?

Interpreting p-values

Community
  • 1
Glen_b
  • 257,508
  • 32
  • 553
  • 939