2

Bayesian models are supposedly well equipped to deal with high-dimensionality problems, and can handle sparse data well, too. But suppose I've created a model that estimate more parameters than there are data points. Are there tricks to deal with this?

Brash Equilibrium
  • 3,565
  • 1
  • 25
  • 43
  • 2
    This is more than possible; if you construct a hierarchical model, this is common practice. Are you able to share more information about the problem you are trying to solve? – Sycorax Nov 19 '13 at 19:36
  • 1
    I think your first sentence needs a citation! As for the second, are these parameters identified from a regression model, network estimation, neural nets... ??? If your problem can be stated as a simple regression problem then, aside from the mere convenience of the interpretation of Bayesian parameters, why would you not prefer a ML technique like LASSO for this problem? – AdamO Nov 19 '13 at 19:41
  • user777: complex model that combines informant reports on dyad states in a tournament graph. There are ~1400 dyads among 80 groups. Within a group, there are only ~3 informants (# of informants and nodes within each group varies). There are ~3000 non-missing informant reports. Also estimating 3 types of informant error from 200 informants. I must estimate criterion dyad states, group-level parameters for the probability of each possible dyad state, and population-level parameters for the probability of each possible dyad state. The same goes for the informant error rates. p gets BIG quickly. – Brash Equilibrium Nov 19 '13 at 19:51
  • user777: But under what conditions is it sensible? Does it depend on the research question? – Brash Equilibrium Nov 19 '13 at 19:59
  • By imposing a prior on sets of the parameters the effective degrees of freedom in the model is reduced. E.g. if we estimate a normal mean for each observation, but the hyperprior is $\mathcal N(0,1)$, it's unlikely we'll pull out a parameter that exactly equals the observed value of some extreme point. – conjectures Nov 20 '13 at 06:54

1 Answers1

1

You say that there are several group-level parameters that you are interested in estimating. That sounds exactly like a hierarchical problem. Do some reading on hierarchical models and the key assumptions (like exchangability), and I think you'll find that your Bayesian background and this problem will easily fit into this paradigm.

The classic example of hierarchical modeling is analyzing data on many students within several schools. We'd expect the process of students' learning to be similar generally, but that better schools would have students with higher achievement, that is, that there is correlation at the student level because each student is not entirely independent of other students in the same school. Accounting for the features of schools that make some better and others worse will help identify salient feature of education, and account for the systematic variation at the student level.

More generally, I would suggest fitting simpler versions of the model and then increasing complexity and the number of parameters gradually. It's much easier to wrap one's mind around, and you will probably learn interesting things about your data along the way which will help you to fit more complex, more realistic models.

Ultimately, what research design is sensible always depends on the research question, but from your description, it sounds like this problem is amenable to a hierarchical approach.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
  • Thanks. In fact, I have specified a hierarchical model. I am just miraculously unaware that such models avoid underdetermination problems. Is it also true that, when a model is undetermined, you should avoid improper priors? – Brash Equilibrium Nov 19 '13 at 21:03
  • All improper priors are admissible so long as they result in proper posterior distributions. But including prior information, even something as simple as a lower bound, can dramatically improve the specificity of the results, that is, reduce posterior variance. – Sycorax Nov 19 '13 at 21:08