13

It seems that there is an ongoing debate within the Bayesian community about whether we should be doing Bayesian parameter estimation or Bayesian hypothesis testing. I'm interested in soliciting opinions about this. What are the relative strengths and weaknesses of these approaches? In which contexts is one more appropriate than the other? Should we be doing both parameter estimation and hypothesis testing, or just one?

sammosummo
  • 743
  • 3
  • 13
  • 2
    Parameter estimation and hypothesis testing are *different* things. I never heard of such debate and I don't know what it would be about? It's like you asked if it is better to eat a dinner, or go for a swim instead. – Tim Nov 17 '16 at 12:23
  • Sorry, but that analogy doesn't work at all. There are definitely some circumstances where either could be applied and definitely a debate about when one or the other is more appropriate. For example, here Kruschke argues that estimation can be used for scientific inference and hypothesis testing (http://www.indiana.edu/~kruschke/BEST/BEST.pdf), but a cursory glance on this site reveals several arguments why this is not appropriate, and that formal hypothesis testing (e.g., via BF, DIC) should be done instead. I can find several more examples of this if necessary. – sammosummo Nov 17 '16 at 12:31
  • 1
    No, he does not make such argument. He shows *how* to estimate Bayesian t-test. If you need to estimate parameter, then you need to estimate parameter, if you need to test a hypothesis, then you need to test a hypothesis, you do not use them interchangeably. – Tim Nov 17 '16 at 12:34
  • OK. I understand that parameter estimation != hypothesis testing. However, that paper is clearly advocating parameter estimation INSTEAD of the traditional t-test, i.e., a hypothesis test. I'm asking in what situations is it more appropriate to use parameter estimation over Bayesian hypothesis testing. – sammosummo Nov 17 '16 at 12:44
  • No, he shows how to estimate Bayesian t-test (that is a hypothesis test). If you want to estimate a parameter, you won't use t-test for it. The fact that you estimated your parameter to be 0.5 does not mean that it is, or isn't significantly different from zero. – Tim Nov 17 '16 at 12:51
  • 1
    The paper is called "Bayesian estimation supersedes the t test". "Supersede" means "in the place of". Ergo, use Bayesian estimation in the place of (instead of) a t test. – sammosummo Nov 17 '16 at 13:32
  • No, its Bayesisn t-test that "supersedes" the frequentist t-test, the paper says nothing about estimation replacing hypothesis testing. – Tim Nov 17 '16 at 13:38
  • 2
    @sammosummo Are you thinking of something like [this Kruschke paper](http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPScorrected.pdf)? – Ian_Fin Nov 17 '16 at 14:09
  • 1
    @Ian_Fin Yes that's exactly what I was thinking about, thank you. I should've checked Kruschke's other publications! I know that he, like Andrew Gelman, is strongly pro estimation and thought I might get more balanced arguments from Cross Validated. – sammosummo Nov 17 '16 at 18:04
  • 1
    In light of the good answers this question has drawn, I think this question should remain open. Perhaps it should be made CW; I don't have a position on that at the moment. – gung - Reinstate Monica Nov 17 '16 at 21:24
  • @Tim I am puzzled by your comments above. The BEST paper definitely *does* claim that (Bayesian) estimation is (usually) better than (Bayesian) testing. It's right there in the paper in the Intro, Discussion, and Appendices, just search for "Bayes factor" and read on. E.g. `Bayesian estimation, with its explicit parameter distribution, is not only more informative than Bayesian model comparison, Bayesian estimation is also more robust`, etc. etc. As OP said, even the title of the paper directly hints that estimation is better than testing. Kruschke wrote other papers arguing the same point. – amoeba Jan 10 '17 at 17:44
  • As this comment thread seems still to be active, please see clarifications at (1) this blog post http://doingbayesiandataanalysis.blogspot.com/2016/12/bayesian-assessment-of-null-values.html and (2) this article https://osf.io/dktc5/ also referenced in my answer below. – John K. Kruschke Jan 11 '17 at 00:02

3 Answers3

9

In my understanding, the problem is not about opposing parameter estimation or hypothesis testing that indeed answers different formal questions but more about how science should work and more specifically what statistical paradigm should we use to answer a given practical question.

Most of the time, hypothesis testing is used : you want to test a new drug, you test $H_O:$ "it effect is similar to a placebo". However, you can also formalize it as: "what is the range of probable effect of the drug ?" which leads you to inference and particularly interval (hpd) estimation. This transposes the original question in a different but maybe more interpretation prone manner. Several notorious statisticians advocate for "such a" solution (e.g. Gelman see http://andrewgelman.com/2011/04/02/so-called_bayes/ or http://andrewgelman.com/2014/09/05/confirmationist-falsificationist-paradigms-science/).

More elaborated aspects of Bayesian inference for such testing purpose includes:

peuhp
  • 4,622
  • 20
  • 38
  • 1
    (+1) Thanks for connecting to our paper! I was wondering whether to mention this aspect... – Xi'an Nov 17 '16 at 14:02
  • 1
    +1 but it might be good to add some links to people (unlike Gelman) advocating *against* Bayesian estimation and in favour of Bayesian hypothesis testing. I have some links in my answer to http://stats.stackexchange.com/questions/200500. EJ Wagenmakers is I think one person who is very much in the Bayesian testing camp. See [Why hypothesis tests are essential for psychological science: A comment on Cumming](http://ejwagenmakers.com/inpress/MoreyEtAlPsychScienceInPress.pdf) and possibly his other papers. – amoeba Jan 10 '17 at 15:37
  • I found your answer to the previous question before I asked this one. It is an excellent answer (and excellent question) and both of them completely supersede mine. – sammosummo Jan 10 '17 at 17:53
  • I think peuhp meant "famous statisticians" not "notorious statisticians". But maybe not! :-) Anyway, if people follow peuhp's link to the posterior predictive check advocated by Gelman and Shalizi, then people should also consider the comments on that article, one of which is here: http://www.indiana.edu/~kruschke/articles/Kruschke2013BJMSP.pdf – John K. Kruschke Jan 11 '17 at 00:25
9

In complement to peuhp's excellent answer, I want to add that the only debate I am aware of is whether or not hypothesis testing should be part of the Bayesian paradigm. This debate has been going on for decades and is not new. The arguments against producing a definitive answer to the question "is the parameter $\theta$ within a subset $\Theta_0$ of the parameter space?" or to the question "is model $\mathscr{M}_1$ the model behind the given data?" are many and, in my opinion, compelling enough to be considered. For instance, in a recent paper, as pointed out by peuhp, we argue that model choice and hypothesis testing can be conducted via an embedding mixture model that can be estimated, the relevance of each model or hypothesis for the data at hand being translated by the posterior distribution on the weights of the mixture, which can be seen as an "estimation".

The traditional Bayesian procedure for testing hypotheses is to return a definitive answer based on the posterior probability of the said hypothesis or model. This is formally validated by a decision-theory argument using Neyman-Pearson's $0-1$ loss function, which penalises all wrong decisions with the same loss. Given the complexity of the model choice and hypothesis testing settings, I find this loss function much too rudimentary to be compelling.

After reading Kruschke's paper, it seems to me that he opposes an approach based on HPD regions to the use of a Bayes factor, which sounds like the Bayesian counterpart of the frequentist opposition between Neymann-Pearson testing procedures and inverting confidence intervals.

Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575
4

As previous respondents have said, (Bayesian) hypothesis testing and (Bayesian) continuous-parameter estimation provide different information in response to different questions. There may be some occasions in which the researcher really needs an answer to a test of a null hypothesis. In this case, a carefully conducted Bayesian hypothesis test (using meaningfully informed, non-default priors) can be very useful. But all too often null-hypothesis tests are "mindless rituals" (Gigerenzer et al.) and make it easy for the analyst to lapse into fallacious "black and white" thinking about presence or absence of effects. A preprint at OSF provides an extended discussion of frequentist and Bayesian approaches to hypothesis testing and estimation with uncertainty, organized around this table: enter image description here You can find the preprint here: https://osf.io/dktc5/

John K. Kruschke
  • 2,153
  • 12
  • 16