5

This may be an obvious question: what do you do when the AIC or BIC choose a model that makes no sense in reality.

I'm looking at data for predictors of hospital selection, and things like a spouse's age are on the right-hand side of the lowest-BIC regression, whereas more obvious things like wealth and education are not.

The BIC of the "best-fit" model is significantly different from the BIC of a more "logical" model, using Rafferty (1995)'s criteria for strength of evidence.

So, my basic question is: do I blindly follow the BIC or do I add in things that I think should be in the equation, even though they'll raise the BIC?

EDITED TO ADD: the difference in the BICs of the "best fit" vs "logical" models is 14 in favor of the best-fit model. The difference in the AICs, however, is 2, in favor of the "logical" model

TheChainsOfMarkov
  • 221
  • 1
  • 5

1 Answers1

6

It's not an obvious question at all! In fact, I think there may be some disagreement even among statisticians.

My view is that you should never let the computer do your thinking for you. Don't blindly accept anything. However, don't blindly reject anything, either. My favorite professor in grad school, Herman Friedman, used to say "if you're not surprised, you haven't learned anything". Well, here you are surprised. Why?

One clear possibility from your description is collinearity. The three variables you mention (wealth, age, education) are related. When there is collinearity, weird things can happen.

If you add more details of the situation and the model, someone (either me or someone else) may be able to offer more insight.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • The model is a latent-class, conditional logit model, where the RHS has variables of two types: alternative (hospital)-specific variables and individual-specific variables. I have a dataset of observed hospital choices for a few thousand folks, as well as data on their potential choice set (all the hospitals they could have chosen). The goal in this question is deciding which individual-specific variables should be included in the model. I'm using the `gmnl` package in `R` for analysis – TheChainsOfMarkov Oct 29 '17 at 15:01
  • One additional point. Hand-calculating VIF for wealth compared to the other regressors appears to give me VIFs that are < 2. – TheChainsOfMarkov Oct 29 '17 at 15:02