0

Consider a standard ordinary linear regression model with $n$ observations and $K>n$ possible regressors (i.e., more regressors than observations).

You know the model is linear but you don't know the relevant regressors. How to proceed with model selection when $n$ is small (i.e., preventing cross-validation)?

mrb
  • 824
  • 8
  • 17

2 Answers2

0

The standard solution is to use stepwise regression. Either forward or a bidirectional method that starts with a single predictor.

I must stress in giving this answer that I am taking your statement that 'you know the model is linear' at face value. If it is not true, then all the normal reasons why stepwise regression is a bad idea come into play.

Tim
  • 3,255
  • 14
  • 24
  • many thanks but could you, please, complete your answer by listing/pointing to the "normal" reasons. – mrb Dec 11 '17 at 03:04
  • Read SmallChess's answers. Also, don't forget to indicate that your question has been answered, as otherwise you are just using up goodwill, which is finite. – Tim Dec 14 '17 at 23:37
0

What you're asking for is a common scenario in feature selection. Stepwise methods are simple to run (only a few R commands), but it's also a greedy method. I don't think it's appropriate for me to repeat advantages/disadvantages for stepwise regression, please read:

Please note the R-square, F-statistic etc in the link all assume linear model is appropriate for your data, otherwise whatever you get might not make sense. Stepewise methods are generally don't perform very well practically, but it's certinaly better than random guessing.

Many statisticians prefer Lasso regression. You have a shinrkage parameter that you can use to control the lasso effects. The first link has the details.

SmallChess
  • 6,764
  • 4
  • 27
  • 48
  • Hi, being the regressors more than the obs, the lasso procedure is of no help here, isn’t it? – mrb Dec 15 '17 at 03:11