4

I have run 2 Bayesian regression models and would like to compare the posterior samples of a parameter that is common to both models.

For example, if model A is

$y=\alpha + \beta_1x_1$

and model B is

$y=\alpha + \beta_1x_1 + \beta_2x_2$

(This is just an example for a difference between models, other differences could be the priors used, hierarchical vs non-hierarchical regression, ...)

what would be the best way to compare the two posterior samples for $\beta_1$ from models A and B, including situations where both posterior samples vary around 0?

Guido Biele
  • 75
  • 4
  • Depends on what do you mean by compare. What do you mean ? what is your purpose ? – peuhp May 30 '16 at 09:55
  • The broader context is that I want make a statement if the regression coefficients from two models can be considered being practically the same or not. One could for example look at the differences or ratio of the parameters. A naive approach would e.g. be to generate a new posterior by calculating the sample by sample difference or ratio. But that does not seem right to me (and generates weird posteriors if coefficients posteriors are around 0). An alternative way would be to do a Bayesian bootstrap, but that does seem to underestimate the uncertainty about the difference. – Guido Biele May 30 '16 at 12:08

1 Answers1

2

While you use the same notation $\beta_1$ in both models, the parameters $\beta_1^1$ for the first model and $\beta_1^2$ for the second model are different parameters and hence cannot be compared in a Bayesian manner. (Take for instance the extreme case when $X_2=X_1$.) The models can be compared by a Bayes factor, for instance, but this is another issue.

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • "the parameters $β_1^1$ for the first model and $β_1^2$ for the second model are *different parameters* and hence cannot be compared in a Bayesian manner" — I don't think that logic is right. Within model B, $β_1$ and $β_2$ are different parameters, but I hope you agree they can be compared in a Bayesian manner. Rather, the problem here is trying to directly compare $β_1^1$ and $β_1^2$ without having assumed enough to determine their joint distribution. – Kodiologist May 30 '16 at 12:17
  • They cannot have a joint distribution: either model 1 or model 2 is the "right" model, they cannot be "right" together. – Xi'an May 30 '16 at 12:20
  • Yes, unless you combined them into a mixture model or something. – Kodiologist May 30 '16 at 12:22
  • I really should have added more context. In my application I know that one model is superior to the other (I don't know about right ...). Still, the relevant research literature predominantly uses the inferior model. (I might add that many know that they should probably use the superior model, but are not necessarily convinced that it makes a noticeable difference). My question then is how can I quantify the difference between the posterior of the coefficients from the inferior and superior model. – Guido Biele May 30 '16 at 12:31
  • @GuidoBiele It sounds like what you might want is just to put plots of the posterior distributions of the two parameters side-by-side. – Kodiologist May 30 '16 at 12:35
  • @Kodiologist. Yes, I've done that. Still, to communicate the comparison in a paper it would be useful to describe the average difference or ratio of the coefficients and the associated highest density interval. I've looked into different ways doing this (which make intuitively more or less sense to me), but I'd like a better justification than my intuition. – Guido Biele May 30 '16 at 12:43
  • 1
    @GuidoBiele In that case, I agree with Xi'an that it doesn't make sense to do that. You need a joint distribution to do that, and you don't have one so long as your models are competitors; that is, that only one is true. – Kodiologist May 30 '16 at 12:46
  • @Xi'an & Kodiologist: I agree that a joint distribution would be best, but I don't think one can say that the comparison makes no sense. Let's take a a slightly different example: You have two posterior distributions for a regression coefficient estimated with an identical model but with different data. I think it should be possible to compare these 2 posteriors (in a Bayesian or non-Bayesian way). – Guido Biele May 30 '16 at 13:16
  • 1
    A comparison that makes sense [to me] is through the predictives but obviously this does not tell you how the $\beta_1$'s differ... – Xi'an May 30 '16 at 13:18
  • I assume with predictives you mean the y_hat. Is this correct? (I recognize "predictives" from "posterior predictive distribution".) – Guido Biele May 30 '16 at 13:30
  • In this [paper](http://arxiv.org/abs/1412.2044), we use a mixture for testing (whether or not $\beta_2=0$), which means producing a posterior on $\beta_1$ under the mixture model, which can serve as a reference against both other posteriors on $\beta_1$ f you do not want to run the test. – Xi'an May 30 '16 at 13:42