9

BIC penalizes based on the number of parameters. What if some of the parameters are some sort of binary indicator variables? Do these count as full parameters? But I can combine $m$ binary parameters into one discrete variable that takes values in $\{0,1,...,2^m-1\}$. Are these to be counted as $m$ parameters or one parameter?

highBandWidth
  • 2,092
  • 2
  • 21
  • 34

1 Answers1

3

It is partly because of this imprecision in the "number of parameters" in BIC that DIC (the deviance information criterion) introduced an effective number of parameters as $$ p_D(x) = \mathbb{E}[D(\theta)|x] - D(\mathbb{E}[\theta|x]) $$ where $$ D(\theta)=-2\log f(x|\theta) $$ and $$ \text{DIC}(x) = p_D(x) + \mathbb{E}[D(\theta)|x] $$ Note that $p_D(x)$ is then data-dependent. (As discussed there, DIC also has problems of its own!)

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • So I am slightly confused. I thought BIC was an approximation of $E[log P(y|Model)] = \log(\int P(y|\theta)P_{model}(\theta)d\theta)$, which can be calculated from the MCMC simulations. Why then would we calculate DIC? – highBandWidth Jan 25 '12 at 22:44
  • 1
    Yes, BIC is an approximation of the marginal likelihood. However, it is only an approximation that converges to the "truth" when the sample size grows to infinity. It is therefore not directly Bayesian (does not use the prior, for one thing!) and completely unrelated to MCMC (where the approximation is of a Monte Carlo type: if I increase the number of simulations, the approximation improves). DIC is deemed to be more Bayesian by many (incl. B. Carlin and D. Spiegelhatler) – Xi'an Jan 26 '12 at 14:46
  • I guess my question was, is the DIC an approximation of the marginal model likelihood too? I guess I should read about it myself, but since we were discussing it, I thought explaining this would make the answer more complete. Thanks! – highBandWidth Jan 26 '12 at 17:11