1

I would like to select the best model for predicting breast cancer risk, specifically, it is the comparisons between weight/BMI/height, as other covariates remain the same in all the models. But I got opposite results for nested model selection by using BIC and LRT. Say that the P-value of LRT is <0.05, but the BIC of the richer model is larger than the less rich model, and the delta BIC is larger than 2, which suggests positive evidence.

Q1: Should I use BIC for nested model comparison? Can BIC penalize the overfitting issue adequately for the richer models?

Q2: Can I use BIC for non-nested model selection and LRT for nested model selection? For example, using BIC to select the best model among weight-only/BMI-only/height-only models; using LRT to select the best model between weight+BMI model Vs BMI-only model to see if weight added additional significant information to the BMI-only model.

I know there are many posts about model selections using BIC/AIC/LRT. But none of them really solve my question.

Padmanabha Chowdhury
  • 137
  • 8
Zhoufeng
  • 101
  • 8
  • See the answer here: https://stats.stackexchange.com/questions/198799/aic-versus-likelihood-ratio-test-in-model-variable-selection. It concerns AIC rather than BIC but the answer is the same. – Spätzle Aug 12 '21 at 04:15
  • I've read that post too, and I raised a question there. I did not get the answer to my question... – Zhoufeng Aug 12 '21 at 04:18
  • It still depens on your purpose: If you're just looking to find a better explaining model, then surely you should go by the BIC. – Spätzle Aug 12 '21 at 04:21
  • I'm looking for a better predictor, BMI/weight. In all the fitted models, the weight-only model had the smallest BIC. what is confusing to me is that the BIC of the BMI-only model is 3-unit smaller than the BMI+weight model, which means BMI is a better predictor than BMI and weight combined. However, the LRT for BMI+weight Versus BMI is significant, which means weight added additional information to the BMI model. How would you explain these results? – Zhoufeng Aug 12 '21 at 04:45

1 Answers1

1

by definition, BMI and weight are related ($BMI=w/h^2$). Taking them together does not add a significant amount of information to your model, hence the unimpressive BIC difference. In large sample sizes, you would almost always find the additional variable significant - that's a key problem of the traditional significance testing and specifically looking only at the p-value.

Recall that p-value is not "the probability that $H_0$ is wrong" but rather "the probability for rejecting $H_0$ in favor of a powerful alternative $H_1$", and then think of how to compute the power of $H_1$.

Nah, I'm kidding. You don't really need to compute it. Just keep in mind that a p-value does not say anything by itself and that with great sample sizes comes great responsibility for being cautios in interpreting the results.

Spätzle
  • 2,331
  • 1
  • 10
  • 25
  • Thanks for your answer. If the weight model is the best based on BIC. And the BIC of weight+BMI model is smaller than the BMI model. Would that make sense to compare the goodness of fit of the two models based on LRT? The result is that the p value is less than 0.05 which could mean weight added additional information to BMI model, If we assume the importance of LRT.. – Zhoufeng Aug 12 '21 at 13:12