2

Suppose I have observed 3 realizations of two non-negative, integer random variables $X$ and $Y$. Nothing is known about their underlying distribution. The results were $x = \{4,~8,~2\}$, and $y = \{22,~11,~8\}$. Hence, $\overline{x} = 14/3$ and $\overline{y} = 41/3$, for a sample mean difference of $9$.

My question is: is there any way to conduct a meaningful statistical hypothesis test in order to decide whether $H_0: E(X) = E(Y)$, can be rejected?

My first idea was to use a two-sample t-test or Welch-test, but I guess the sample size is way too low for that. Is there anything one can reasonably test?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Martin
  • 179
  • 1
  • 1
  • 9
  • 4
    You may find the following thread informative: [is-there-a-minimum-sample-size-required-for-the-t-test-to-be-valid](http://stats.stackexchange.com/questions/37993/). – gung - Reinstate Monica Jun 25 '13 at 14:52
  • What makes you say the sample size is too low? The big problem is that you can't check the assumptions, but a two sample t-test can actually be performed with two observations in each sample - or indeed, if you're prepared to assume equality of variance, with even as little as 3 observations total! – Glen_b Jun 26 '13 at 02:28

2 Answers2

6

There are potentially a number of ways of testing if these two samples differ, but all will probably have low power. You could use a t-test, but its validity will depend on whether the underlying populations are normally distributed and have equal variances. With so few data, you really can't check that very well so you have to rely entirely on prior knowledge (of which you say you have none) and the assumptions you are willing to make. Given that your variances are $9.3$ and $54.3$, I would not want to make the assumption of equal variances (although, again, with so few data they actually could be), so the Satterthwaite-Welch correction seems appropriate. If you weren't willing to assume the populations were exactly normal (since the central limit theorem cannot cover you with samples this small), you could use the Mann-Whitney U test. As it happens, that test gives a lower p-value ($.12$) than the corrected t-test ($.16$). The question then, is what you want to conclude from these results. My opinion is that using a rigid $.05$ cutoff is typically not appropriate (see my answer here: When to use Fisher and Neyman-Pearson framework?); so I would say this result is somewhat ambiguous, but you might find it does provide some evidence against the null, depending on how plausible the null is a-priori.

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • 1
    how did you compute your Mann-Whitney p-value? did you use the normal approximation? adjust for ties? etc.? The exact permutation test gives a p-value of 0.10 and since the MW test is a permutation test it should give a p-value of 0.10 as well (but I could see a small difference coming from an approximation rather than an exact test). – Greg Snow Jun 25 '13 at 19:55
  • @GregSnow, I just typed the data into R & ran `wilcox.test(x, y)`. – gung - Reinstate Monica Jun 25 '13 at 19:58
  • @Greg The M-W is not a permutation test of the mean, so it should be no surprise that it gives a (slightly) different p-value. With some of the permutations, such as $((2,4,22), (8,8,11))$, the order of the means ($28/3, 27/3$) differs from the order of the medians ($4, 8$) or the rank sums ($9, 12$). How you adjust for ties can make a substantial difference, too, depending on what test statistic is used in the permutation test. Gung: talking about "power" may be a little dicey when no alternative hypothesis is clearly formulated. – whuber Jun 25 '13 at 20:16
  • @whuber, There are only 20 possible combinations for splitting 6 observations into 2 groups of 3 each, so any p-value from an exact permutation test will be a multiple of 0.05. Since the data has 2 8's, 1 in each group that means that any permutation that switches the 2 8's will be equivalent giving a p-value of 0.10 (though it should be doubled to 0.2 for a 2-sided test) for any statistic that measures location, whether it be mean, median, minimum, sum of ranks, 42nd percentile, etc. A permutation test based on ratio of variances could give a 0.05 for a 1-sided test. – Greg Snow Jun 25 '13 at 20:45
  • @gung, ok, that function uses the normal approximation since there is a tie. – Greg Snow Jun 25 '13 at 20:46
  • @Greg That much is clear--the problem is so small it is easy to enumerate all the possibilities. How ties are dealt with becomes important since they affect an appreciable fraction of all permutations. I agree that the p-value should be a multiple of 0.05, but the MW p-value needn't be the same as a p-value for a permutation test based on means. The fact remains, though, that since one sample dominates another and there is a tie in the data, the MW and permutation test p-values will (somewhat accidentally) be the same in this particular circumstance. – whuber Jun 25 '13 at 20:52
  • @whuber, not accidentally at all if the exact test is used rather than an approximation. See: Different Outcomes of the Wilcoxon—Mann—Whitney Test from Different Statistics Packages Reinhard Bergmann, John Ludbrook & Will P. J. M. Spooren Journal: The American Statistician, Volume 54, Issue 1, February 2000, pages 72-77 – Greg Snow Jun 25 '13 at 21:20
  • 1
    @gung I think the big problem with the Mann-Whitney on this tiny a sample size (3,3) is that - no matter how vastly separated the samples are, and even without any ties - you simply can't get a p-value below 0.1 (two tailed). At least a suitable parametric test (if its assumptions could be argued to be reasonable) has the *possibility* of reaching an interesting p-value. One needs to bring some assumptions (not necessarily normality) to this small a sample, and try to justify them. – Glen_b Jun 26 '13 at 02:33
  • Many thanks for the active discussion so far! I'm currently at a conference, but I'll try to have a closer look at everything as soon as possible. – Martin Jun 27 '13 at 18:41
4

You say that nothing is known about the distribution, but you also say that the values are non-negative integers, so that tells you something, can you learn more about the theory?

An exact permutation test on your data gives a p-value of 0.10 and does not require assumptions about the distribution (just testing that it is identical for the 2 groups).

If your data represent counts (non-negative integers) then a Poisson model may be appropriate (which gives a p-value less than 0.001), but make sure that the assumptions are reasonable.

As @gung mentions, there are many ways to look at this and which is best is going to depend on the source of the data and the science that underlies it. learning about the data, talking to the person that gave you the data, talking to other experts, etc. is going to be of the most benefit.

If you really cannot learn anything more about the underlying distribution or approximations to it then you may have to resort to SnowsCorrectlySizedButOtherwiseUselessTestOfAnything (a function in the TeachingDemos package for R) which will give a p-value without requiring any assumptions about your data. But note that that function is considered less useful than its documentation.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159