14

Would you consider the HBM vs EB to be two alternatives in which the hyperparameters are "in the game" of being sampled/estimated/etc.? There is clearly a connection between these two.

Would you consider HBM more "fully Bayesian" than EB? Is there some place where I can see what are the differences between being "fully Bayesian" and the other alternatives?

Thanks.

Firebug
  • 15,262
  • 5
  • 60
  • 127
singelton
  • 652
  • 5
  • 13
  • 1
    For a discussion on what "fully Bayesian" and "Empirical Bayes" mean see the answers in ["“Fully Bayesian” vs “Bayesian”"](http://stats.stackexchange.com/q/31849/10525). –  Aug 14 '12 at 15:53
  • thanks Procrastinator. I still would like to hear a response about the relationship to hierarchical Bayesian models, if possible. – singelton Aug 14 '12 at 15:57
  • 2
    You can find this in the wikipedia entry [Empirical Bayes method](http://en.wikipedia.org/wiki/Empirical_Bayes_method) : "empirical Bayes may be viewed as an approximation to a fully Bayesian treatment of a hierarchical model wherein the parameters at the highest level of the hierarchy are set to their most likely values, instead of being integrated out". –  Aug 14 '12 at 15:59

1 Answers1

11

I would say that HBM is certainly "more Bayesian" than EB, as marginalizing is a more Bayesian approach than optimizing. Essentially it seems to me that EB ignores the uncertainty in the hyper-parameters, whereas HBM attempts to include it in the analysis. I suspect HMB is a good idea where there is little data and hence significant uncertainty in the hyper-parameters, which must be accounted for. On the other hand for large datasets EB becomes more attractive as it is generally less computationally expensive and the the volume of data often means the results are much less sensitive to the hyper-parameter settings.

I have worked on Gaussian process classifiers and quite often optimizing the hyper-parameters to maximize the marginal likelihood results in over-fitting the ML and hence significant degradation in generalization performance. I suspect in those cases, a full HBM treatment would be more reliable, but also much more expensive.

Dikran Marsupial
  • 46,962
  • 5
  • 121
  • 178
  • 6
    +1 for **EB ignores the uncertainty in the hyper-parameters**. Also, Bayesian fundamentalists consider EB *anti-Bayesian* because using the data for estimating the prior is *blasphemous*. –  Aug 14 '12 at 16:34
  • 4
    Evidently I am a not a fundie Bayesian then! HBM seems to me to be the right thing to do, provided it is actually computationally feasible, at the end of the day you need to be pragmatic to a degree (after having bought the biggest computer available ;o). – Dikran Marsupial Aug 14 '12 at 16:39