9

While learning frequentist linear regressions, one thing the professors always talked about was about the number of degrees of freedom, I never saw this expression in a bayesian book though. Perhaps because bayesian methods don't need this number to infer things like variance and such?

My question is: is the number of degrees of freedom equals the number of parameters in an hierarchical bayesian model and if it's not, is there something equivalent one can calculate? In particular, I'm interested in when a model is overidentified in a hierarchical framework.

For example, if I have 1000 observations and about 10 possible competing models with about 100 parameters each, if mix them all in an hierarchical model using, for example, trans-dimensional MCMC/Bayes factor, the will I have an overidentified model?

My intuition says that it's possible that it won't, although the total number of parameters is greater than the number of observed parameters.

random_user
  • 760
  • 1
  • 5
  • 20
  • In what situation would you need DFs for a Bayesian analysis? We don't use test statistics, since (IMO) the main strength of Bayesian stats is summarizing posterior beliefs rather than contrived null-hypothesis-significance-testing. – AdamO Jan 12 '18 at 16:53

2 Answers2

3

At least from a theoretical point of view, identifiably is not important from a Bayesian perspective. If the data is not informative about some parameters under the model then the posterior of those parameters will just be highly influenced by the prior.

From a practical point of view if the posterior is broad then approximate methods such as MCMC will take longer maybe much longer to run.

Another practical problem is that if you have a large parameter space and little data as it sounds like you do then the results, if you can manage to compute them, are likely to be very sensitive to prior specification.

David Rohde
  • 31
  • 2
2

There is literature on Bayesian inference on over-identified models (e.g. Gelfand and Sahu, 1999. J. Amer. Statist. Assoc. 94:247-253), that is when the number of estimands in a model exceeds the number of (independent) observations. If priors are proper, the posterior is proper as well, but Bayesian learning on non-identified parameters depends on how much is learned about items that are identified. Hence, priors are influential, and this may be a serious matter with Bayesian models fitted, say, to DNA data, where the number of unknowns is in the dozens of millions. Caution should be exercised, e.g., in medical genetics.

There is a concept called "effective number of parameters" or neff (see, for example, in the Deviance Information Criterion, or in regression models with shrinkage). In all cases, the neff is at most n.

As in the good all times: the number of independent questions that one can ask from a data set is, at most, n. Hence if you pose n+k questions, k of the answers will b redundant with respect the first n answers.

In short, statistical learning must be imperfect in overidentified models no matter how fancy you are in "regularizing" the model or how eloquent your local Bayesian resident expert is.

Daniel Gianola

Daniel Gianola
  • 21
  • 1