1

I understand adjusted r-squared and AIC can be used to select an ideal model from a group. Higher AIC is worse but higher ar2 is better.

After adding a categorical variable to an OLS model, my adjusted r-squared went up and so did my AIC. Some of the categorical values are associated with a high p-value.

Three questions:

  1. How can I interpret the discrepancy?
  2. I tend to favor including the variable because of the ar2 effect. Is this a fair justification?
  3. Is either metric generally preferred among AR2 and AIC?
Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
John Vandivier
  • 133
  • 6
  • Why are you choosing between these models? Why don't you just have one model that you fit, and that's the model? – Dave Oct 13 '21 at 17:29
  • 1
    See [this thread](https://stats.stackexchange.com/questions/408267) regarding the lack of justification for $R^2_{adj.}$ as a model selection criterion. – Richard Hardy Oct 13 '21 at 17:32
  • Thanks @RichardHardy! @Dave my use case is identifying the most important features (eg forward selection/backwards elimination) – John Vandivier Oct 13 '21 at 17:53
  • [Stepwise model selection is problematic.](https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/) // [Gelman agrees.](https://statmodeling.stat.columbia.edu/2014/06/02/hate-stepwise-regression/) – Dave Oct 13 '21 at 17:57
  • @Dave I have theory-based supervision of the variable pool, I'm not maximizing r-squared, I am performing cross-validation...I think those objections are largely leveraged against some other form of selection. Do you have anything in the way of positive, preferred suggestion? – John Vandivier Oct 13 '21 at 18:01
  • @RichardHardy I read your link but it doesn't seem to have been decisively answered? Oren discussed what I consider to be the standard advice in my learning experience (AR2 is valued for parsimony/complexity minimization/anti-overfit properties of R2 alone). I added a comment based on my limitted knowledge of AIC – John Vandivier Oct 13 '21 at 18:22
  • @Dave When I mentioned (eg forward selection/backward elimination) I was suggesting two procedures, not one procedure of alternating between them. Maybe that is the miscommunication. Do you consider backward elimination to be an example of stepwise selection? This paper and some others draw a bright line between those, while others seem to group them https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5969114/ – John Vandivier Oct 13 '21 at 23:19
  • related: https://stats.stackexchange.com/questions/140965/when-aic-and-adjusted-r2-lead-to-different-conclusions – John Vandivier Oct 18 '21 at 00:23

0 Answers0