12

I am looking for the bayesian counterpart of the two-sample t-test with unequal variances (the Welch test). I am also looking for a multivariate test, like Hotelling's T statistic. References appreciated.

For the multivariate case, suppose that we have $(y_1,\cdots,y_N)$ and $(z_1,\cdots,z_N)$, where $y_i$ (resp $z_i$) is a shortcut for a sample mean, sample standard deviation and number of points. We can assume that the number of points is constant across the whole dataset, the standard deviation the same for all $y_i$ (resp $z_i$) and that the sample means of the $y_i$ (resp $z_i$) are correlated. If you plot the sample means, they follow each other and by connecting them, you get a smooth varying function. Now on some parts the $y$ function agrees with the $z$ function, but on others it doesn't, because $\frac{mean(y_i)-mean(z_i)}{std(y_i)+std(z_i)}$ becomes big. I would like to quantify this statement.

yannick
  • 762
  • 4
  • 12

2 Answers2

11

John Kruschke has developed a Bayesian routine that is meant as a drop in replacement for the two-sample t-test. The routine is called BEST (Bayesian Estimation Supersedes the T-test) and is described here. I also made an online javascript version that runs in the browser available here.

Rasmus Bååth
  • 6,422
  • 34
  • 57
6

While you can do this in a Bayesian way, have you considered whether it would actually be better to estimate the difference in the means rather than test whether they are different? This is what Andrew Gelman frequently recommends. I can imagine some possible reasons for wanting to do hypothesis testing, but I don't think they're that common.

I don't think you need something like a t-test, because you can estimate the standard deviation well because you said the groups have very similar standard deviations.

If that's the case then I think this link should be what you need. It shows how to estimate a difference in means or do a hypothesis test (though I don't recommend this). You could also take a look at the part they reference in bolstad's book (you can find electronic copies online). Its possible to incorporate estimating the variances as well but it's more complex, so I suspect you're better off incorporating the prior information you have about the variances in a naive way (for example, using the unbiased Stdev estimator on each of the sets and then averaging them and pretending those are your 'known' stdevs).

John Salvatier
  • 4,032
  • 1
  • 18
  • 28
  • yes, but that leads to another problem. How can you know whether the difference in the means is actually significant? I'd compare it to the sum of the SD of each sample, but that's not very rigorous. – yannick Sep 20 '11 at 05:57
  • @yannick: "significant", statistically or real-world? – Wayne Sep 20 '11 at 15:18
  • @Wayne real-world I suppose. – yannick Sep 20 '11 at 16:14
  • 3
    @yannick: Real-world significance is a domain knowledge problem, not a statistical one. That is, I can tell you that I have some weight data and there is a statistically significant 10 gram difference in mean weights between two groups, at the 95% level, but does that have real-world significance? For a minnow, yes, for adult men, no. If you're talking real-world significance, I'd imagine comparing to SD or determining quantiles would answer your question even if that doesn't seem rigorous and leaves room for someone to disagree with you. – Wayne Sep 20 '11 at 16:32
  • @Wayne Suppose I look at $\frac{m_1-m_2}{s_1+s_2}$, you're saying that the decision to when we can say "significant" effect size is arbitrary? And so is the choice of the link function that would map that quantity to [0:1] ? Aren't there practical things people do? – yannick Sep 20 '11 at 18:17
  • Not sure I understand your latest question. I was just trying to clarify whether your objection to @John Salvatier was because you were looking for a statistical significance test or because you were afraid that saying "a mean difference of X is obviously a significant difference". – Wayne Sep 20 '11 at 18:31
  • let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/1385/discussion-between-wayne-and-yannick) – Wayne Sep 20 '11 at 18:32
  • I think it would help if you added some information about what decision you want information to help you make. – John Salvatier Sep 21 '11 at 15:53
  • @JohnSalvatier I did the modifications – yannick Sep 22 '11 at 08:36