4

I have some potential spline models and I'm trying to use AIC or BIC to choose variables. I'm seeing that AIC is lower when I use all variables than if I exclude any one or two. However, if I exclude three variables, then AIC is lower than if I include all variables. Is this theoretically possible or am I potentially doing something wrong? My concern is that if I do backwards selection with AIC, the optimal model chosen will be all variables since incrementally dropping one will not reduce AIC.

Sorry I can't post a reproducible example as I have a quite complex spline model and the data is proprietary to my company.

Gaurav Bansal
  • 633
  • 1
  • 6
  • 16

1 Answers1

4

Yes, this is definitely possible.

Remember that $$ \mathrm{AIC} = 2k - 2 \log L $$ where $k$ is the number of parameters and $L$ is the likelihood. As you remove features, both $k$ and $L$ decrease, so the $2k$ term decreases and the $-2 \log L$ term increases. Each of those terms is monotonic as you remove features, but their sum need not be, as they change at different rates.

If you can, it's usually better to use held-out or cross-validated likelihoods rather than AIC. That doesn't necessarily avoid the problem you're worried about, though; model selection is just hard.

Danica
  • 21,852
  • 1
  • 59
  • 115