14

I recently happened to read about empirical Bayes (Casella, 1985, An introduction to empirical Bayes data analysis) and it looked a lot like random effects model; in that both have estimates shrunken to global mean. But I have not read it throughly...

Does anyone have any insight about the similarity and differences between them?

Firebug
  • 15,262
  • 5
  • 60
  • 127
anonymous
  • 141
  • 1
  • 3
  • 1
    Empirical Bayes can be used in situations with or without random effects - EB simply refers to Bayesian approaches that estimate, from the data, parameters (sometimes called hyperparameters) of the prior distribution - this is an estimation method whereas random effects models are an approach to modeling correlated data. Perhaps the example you saw involved estimating a random effects model using empirical Bayes and that's why you're connecting the two. – Macro Jun 04 '12 at 12:49
  • 1
    Casella, not Cassella! – Xi'an Jun 04 '12 at 20:13
  • 2
    A major difference is that random effect models are models (including a random effect), while empirical Bayes techniques are inference techniques: e.g., you can run an empirical Bayes estimation on a random effect model... Empirical Bayes methods apply in about every situation where you could use a regular Bayes method, not only to random effect models. – Xi'an Jun 04 '12 at 20:17
  • 1
    Related question: [Unified view on shrinkage: what is the relation (if any) between Stein's paradox, ridge regression, and random effects in mixed models?](http://stats.stackexchange.com/questions/122062) – amoeba Nov 02 '14 at 15:48

1 Answers1

6

There is a really great article in JASA back in the mid 1970s on the James-Stein estimator and empirical Bayes estimation with a particular application to predicting baseball players batting averages. The insight I can give on this is the result of James and Stein who showed to the surprise of the statistical world that for a multivariate normal distribution in three dimensions or more the MLE, which is the vector of coordinate averages, is inadmissible.

The proof was achieved by showing that an estimator that shrinks the mean vector toward the origin is uniformly better based on mean square error as a loss function. Efron and Morris showed that in a multivariate regression problem using an empirical Bayes approach the estimators they arrive at are shrinkage estimators of the James-Stein type. They use this methodology to predict the final season batting averages of major league baseball players based on their early season result. The estimate moves everyone's individual average to the grand average of all the players.

I think this explains how such estimators can arise in multivariate linear models. It doesn't completely connect it to any particular mixed effects model but may be a good lead in that direction.

Some references:

  1. B. Efron and C. Morris (1975), Data analysis using Stein's estimator and its generalizations, J. Amer. Stat. Assoc., vol. 70, no. 350, 311–319.
  2. B. Efron and C. Morris (1973), Stein's estimation rule and its competitors–An empirical Bayes approach, J. Amer. Stat. Assoc., vol. 68, no. 341, 117–130.
  3. B. Efron and C. Morris (1977), Stein's paradox in statistics, Scientific American, vol. 236, no. 5, 119–127.
  4. G. Casella (1985), An introduction to empirical Bayes data analysis, Amer. Statistician, vol. 39, no. 2, 83–87.
cardinal
  • 24,973
  • 8
  • 94
  • 128
Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • 1
    Not completely related, but a little more on the (in)admissibility results [are in this question](http://stats.stackexchange.com/questions/13494). – cardinal Jun 04 '12 at 12:49
  • 1
    I have placed a link to the article I think you are referring to as item (1) under the references, but since Efron & Morris wrote a whole host of articles on related topics during that time period, it's somewhat unclear which one you were actually referring to. I've also tried to adjust some of the formatting and spelling. Please check that I haven't inadvertently introduced any errors and feel free to edit further or rollback any of mine. – cardinal Jun 04 '12 at 13:05
  • 1
    I have placed links to authoritative archives in the post, but, some or all of the articles may be found at other (less stable) sources on the web. – cardinal Jun 04 '12 at 13:07
  • @Cardinal Thanks for adding the references. The baseball example I referred to was in the 1975 JASA article. – Michael R. Chernick Jun 04 '12 at 13:35
  • 1
    Thanks for posting the Efron and Morris article. A reminder of better days, when Don Kessinger, Ron Santo and Billy Williams were playing for the Cubs, and Scientific American still published articles worth reading. – Ringold Jun 04 '12 at 14:21
  • @Ringold Ah but even those great Cub teams like in 1969 fell to the surge of the Amazin Mets. – Michael R. Chernick Jun 04 '12 at 14:29
  • 1
    A very recent monograph by Brad Efron, [Large-scale inference](http://www.amazon.com/gp/product/0521192498/ref=as_li_ss_tl?ie=UTF8&tag=chrprobboo-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0521192498) appeared recently. Despite its title, it is all about empirical Bayes! (See [here](http://xianblog.wordpress.com/2012/02/24/large-scale-inference/) for my review of the book.) – Xi'an Jun 04 '12 at 20:18
  • @Xi'an The book is about large scale inference but the method of choice is empirical Bayes. – Michael R. Chernick Jun 04 '12 at 20:23
  • Would the obvious connection to a mixed effects model be that (to stay with the baseball example) you could have a hierarchical model with a random player effect (e.g. normal random effect on the logit of the probability of a successful bat) and that this would also shrink the estimated latent player effects towards the common mean in a similar manner as an empirical Bayes approach? – Björn Jul 28 '17 at 16:13