3

I am confused on what I have read about the Mann whitney test. We are testing whether our actual data is the same as our projected data. We had been using the t-test until we realized the data might not be normal and we are using ~12 observations per sample. Therefore I started using the Mann Whitney test.

I have read some places that the test is not appropriate unless the variances are equal, and other places have said you can still use it if you don't care if the shapes are identical.

So do I need to test each sample against the other to see if the variances are equal? Do the interpretation of the results differ if the variances are not equal?

Glen_b
  • 257,508
  • 32
  • 553
  • 939
adam
  • 41
  • 1
  • 1
  • 2
  • As far as I know, the Mann Whitney test also assumes homogeneity of variances. Several papers by Vargha and Delaney, and at least one paper by Zimmerman (2004) discuss this (and show the Type I error rate with simulation studies). – Patrick Coulombe Aug 26 '14 at 19:40
  • 1
    From Zimmerman (2004): "The present study discloses that, for a wide variety of non-normal distributions, especially skewed distributions, the Type I error probabilities of both the t test and the Wilcoxon-Mann-Whitney test are substantially inflated by heterogeneous variances, even when sample sizes are equal." (http://files.eric.ed.gov/fulltext/EJ848306.pdf). This paper talks at length about the misconceptions regarding assumptions of the Mann Whitney and Kruskal Wallis tests: http://jeb.sagepub.com/content/23/2/170.short – Patrick Coulombe Aug 26 '14 at 19:43
  • 3
    Adam, The topic of the assumptions of Mann-Whitney/Kruskal-Wallis has been discussed on this site numerous times. You may search and read. In short: you don't _have_ to assume equal distributional shape (which automatically implies equal variance) in the populations. The test tests for the so called stochastic dominance then. If you _do_ assume equal shape, then the test tests for shift of one distribution relative the other. So _you_ choose between the 1st (wider) and the 2nd (narrower) appication of the test. – ttnphns Aug 26 '14 at 21:25
  • @ttnphns: To test for stochastic dominance you *do* have to assume the cumulative distribution functions don't cross. Else you're just testing whether the probability that an observation from one population is greater than an observation from the other population differs from one half. – Scortchi - Reinstate Monica Aug 27 '14 at 10:44
  • @Scortchi, Saying "stochastic dominance" I meant exactly that thing about the probability one half. Was I misusing the term? – ttnphns Aug 27 '14 at 12:52
  • 2
    @ttnphns: Yes - stochastic dominance of $X$ over $Y$ means $\Pr(X>a) \geq \Pr(Y>a)$ for all $a$ (& of course $\Pr(X>a) > \Pr(Y>a)$ for some $a$). Note also that stochastic dominance of $X$ over $Y$ together with stochastic dominance of $Y$ over $Z$ implies stochastic dominance of $X$ over $Z$, whereas $\Pr(X>Y)>\frac{1}{2}$ together with $\Pr(Y>Z)>\frac{1}{2}$ doesn't imply $\Pr(X>Z)>\frac{1}{2}$; dice illustrating the latter situation make nice Xmas presents. – Scortchi - Reinstate Monica Aug 27 '14 at 13:12
  • @Scortchi, thank you a lot. So, what is your opinion - does MW (in its general formulation, i.e. without "equal shapes" or whatever assumptions) tests for the stochastic dominance or for the probability not 1/2? That is: do we _always_ have to assume not crossing cum. distributions in the population, and without this assumption the test is useless? Or not? What can you say? – ttnphns Aug 27 '14 at 13:35
  • 1
    @ttnphns: It tests, in general, whether the probability that an observation from one population is greater than an observation from the other population differs from one half. If you assume the cdfs don't cross that implies stochastic dominance of one over the other. It's often a reasonable assumption - a poison, say, retards the growth of some seedlings more than others, but doesn't advance that of any - & can be informally checked by examining the empirical cdfs. Without that assumption, you can say, well, the probability that an observation &c., but that statement doesn't have all the ... – Scortchi - Reinstate Monica Aug 27 '14 at 14:09
  • ... implications that one might think; in practice I then look at histograms or whatever to see what's going on. – Scortchi - Reinstate Monica Aug 27 '14 at 14:09

2 Answers2

7

The Mann-Whitney test first ranks all your values from high to low, computes the mean rank in each group, and then computes the probability than random shuffling of those values between two groups would end up with the mean ranks as far apart as, or further apart, than you observed. No assumptions about distributions are needed so far.

If you want to make further inferences about difference between medians, you need to assume that the two populations have about the same shape distributions, so the same variances, (even if the medians are different so the two distributions are shifted from one another).

Harvey Motulsky
  • 14,903
  • 11
  • 51
  • 98
  • Thank you for your answer. I am using the U critical value to determine the p value. I am not 100% sure if that means I am making inferences about the difference in medians. When the mann whitney U critical value and such is calculated, does it require inference on medians? (I am using this http://www.real-statistics.com/non-parametric-tests/mann-whitney-test/) – adam Aug 26 '14 at 19:38
  • +1. But... How should one understand `about the same shape distributions...even if the medians are different`. I thought that the assumptions are about populations, not samples. Therefore the shapes there are exactly the same (which involves equal medians). – ttnphns Aug 26 '14 at 21:32
  • 1
    @ttnphns You have a point. I believe Harvey is discussing the *alternative hypothesis,* which is that the two samples come from populations which are shifted (in location) with respect to each other. – whuber Aug 26 '14 at 22:32
  • 1
    @adam The P value is accurate even if you don't assume anything about the shape of the distribution. But then it is asking about the discrepancy in mean ranks. If you want to make an inference about the discrepancy between the medians, you'll need to assume that the shape of the two population distributions is the same, even if the two are shifted (and so have different medians). I corrected my answer to be more clear. – Harvey Motulsky Aug 27 '14 at 01:30
5

We are testing whether our actual data is the same as our projected data.

No you're not. You can answer that at a glance (the data are different). Hypothesis tests are for doing something else.

It's not clear however, that a hypothesis test is a good choice for your problem, and even if it were, I wouldn't use these.

We had been using the t-test until we realized the data might not be normal and we are using ~12 observations per sample. Therefore I started using the Mann Whitney test.

I think a big problem here is that actual-vs-projected should be paired (by the observation that they're the actual and projected of). Consider that if you ignore the pairing that you'd regard both these as equally adequate:

     Actual:   6  9 12 15 16 14 19 20 20 22
Projected 1:  11 12 13 14 15 16 17 18 19 20
Projected 2:  20 19 18 17 16 15 14 13 12 11

However, even taking account of pairing on observation doesn't solve the underlying problem -- that projections may be exactly right on average, but still be terrible. You want the individual projecteds somewhere close to individual actuals (close overall) and a test of means doesn't address that at all.

A better way to measure difference between actual and projected would be something like a sum of squares of error but I don't think you should be trying to test anything. It doesn't really answer the kind of question that it makes much sense to ask (such as "are our projections good enough for our purposes?" or "is this projection approach performing better than that one?" - those kind of questions make sense but aren't hypothesis-testing questions).

I have read some places that the test is not appropriate unless the variances are equal,

It depends on why you're using it. For some uses, you need even more than equality of variance, you need identical shape. For other uses it doesn't matter.

and other places have said you can still use it if you don't care if the shapes are identical.

Basically, that's correct.

The Mann-Whitney U statistic (when scaled by the number of comparisons) estimates the probability that a random observation form one population exceeds a random observation in a second one, irrespective of whether the two are similar in shape.

So it's possible to get a U statistic that shows no indication of a difference (in the sense of being close to what you expect if the null were true) when two distributions are nothing alike.

if you make the additional assumption that the shapes are the same, apart from a possible shift, then the Mann-Whitney is a location test (and quite a good one in many situations).

So do I need to test each sample against the other to see if the variances are equal?

That depends, but what you should do is consider whether a hypothesis test is even the most appropriate approach for your problem (I really don't think it is), and if so, then think about the pairing issue. Those should come first.

Do the interpretation of the results differ if the variances are not equal?

Possibly. If the shapes are the same, you can say something more specific.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • 1. How could you test using sum of squares error? I mean you would have 2 different values, one for actual and one for projected, how would you test if they are significantly different? – adam Aug 26 '14 at 23:46
  • Sorry still not 100% familiar on how to reply to comments on this site. See edited version. Also, . The goal when we test actual v. projected is not to make sure each pair is close but rather the overall average. We have been struggling on what test to use. I know you mention maybe no test is appropriate for the question, but I still see a need to at least be able to say these projections are in line with actuals. We make changes on projections by increasing or decreasingthe overall projectionand want the projection to match on average not necessarily by each pair. – adam Aug 26 '14 at 23:51
  • Projections that match on average can be arbitrarily bad. Consider a good set of projections, and now add a large amount of zero-mean noise (with standard deviation 1000 times as large as the typical difference between good projections and actual). It still has the right *average* (and would not be rejected by a test), but is useless in comparison to the first. – Glen_b Aug 27 '14 at 00:00
  • I wouldn't *test* using sums of squares of error (or mean squared error, perhaps, to scale it appropriately). I'd use that as a basis of comparison between projections, or measure how well a set of projections is doing. Some more clarity about your circumstances (in your question) might help. – Glen_b Aug 27 '14 at 00:03
  • The basis of what Im doing is this. I use a program to run a projection on 12 months ofthe past, andthen compare the projections, without adjustments, to actuals. this program then lets you adjust the probabilities of certain projections (or lets say raise or lower the projections for certain metrics) and you repeat this process until it looks in line. Then we test to make sure it is in line (this is the step we are talking about) and then use the settings for a future forecast. With sum of squared errors, I could compare it to the SSE of the actuals but I have no way of knowing if the – adam Aug 27 '14 at 00:24
  • difference is too big or what. I guess the question is how could you tell if a SSE value is too big compared to another? – adam Aug 27 '14 at 00:24
  • "Too big" would be a judgement that depends on circumstances and who is doing the judging. It's hard to say much without context but it's generally a case of "what would matter" vs "what really wouldn't matter". – Glen_b Aug 27 '14 at 01:04