4

I am reading about 'interaction effects on linear regression' here and came across 'stepwise linear regression'.

There are originally 5 predictors in the model. This means to say that by using the ordinary linear regression, we have $ Y = c + a_iX_i$ where $i = 1,...,5$.

Then it says here: For the initial model, use the full model with all terms and their pairwise interactions.

The succeeding steps involve what it calls 'stepwise linear regression'.

I am confused by this statement. Can anyone please give an insight on what 'stepwise linear regression' is all about? What are its advantages and why does it need to be done?

Maxim
  • 3,164
  • 1
  • 17
  • 25
cgo
  • 7,445
  • 10
  • 42
  • 61
  • 5
    [Stepwise regression](https://en.wikipedia.org/wiki/Stepwise_regression) is something that basically [should not be used](https://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection/20856#20856). – Tim Dec 07 '17 at 15:58
  • 2
    Note that clicking on the `stepwise-regression` tag under your question provides a fairly simple explanation. Also note that most contributors to this site would argue against stepwise regression except in closely defined circumstances. – EdM Dec 07 '17 at 15:58
  • 1
    I am entirely new to this. I have read the links above, though I have not understood it fully. Any reason why it is written in a Matlab documentation (and promoted) to be used? I searched it and the general consensus is this method is controversial. – cgo Dec 07 '17 at 16:17
  • 1
    Even recently, thoughtful writers have allowed that stepwise regression can be useful in some cases. Many (if not all) multiple regression textbooks originally written between 1980 and 2010 or so will describe forward, backward, and all-subsets forms of stepwise regression in detail. Statistical software packages usually provide this capability. – whuber Dec 07 '17 at 16:46

2 Answers2

4

Stepwise Linear Regression is a method by which you leave it up to a statistical model test each predictor variable in a stepwise fashion, meaning 1 is inserted into the model and kept if it "improves" the model. Improve is defined by the type of stepwise regression being done, this can be defined by AIC, BIC, or any other variables. If it worsens the model, the predictor is then taken out. It sort of does some work for you. DON'T SKIP THE NEXT PARAGRAPH!!!!

HOWEVER!!!! this method should be avoided. Nothing wrong with the mathematics, but the logical thinking about how and why each variable should be in a model is not taken into account. What is your reasoning for putting this or that variable in the model? Questions of that nature, to understand our uncertainty about some variable of interest is not accounted for by the stepwise process. These are questions that stepwise regression can't answer, and the variables it's based on to include/exclude variables can't do that either. People loved it before because they could dump 20+ predictor variables in and get an "equation", but they didn't know if it was good or not, and the thinking behind what was in the equation was lost. Otherwise, it's like predicting shoe size by ice cream scoops (totally bogus, even if r^2 = 1.00)

Kunio
  • 343
  • 1
  • 6
  • 3
    +1 but this answer might be a bit harsh. As @whuber notes in a comment, stepwise regression can be useful in some circumstances. Even Frank Harrell allows for "limited backwards step-down variable selection if parsimony is more important than accuracy," but only as the 13th and _last_ step of formulating a final predictive model ([Regression Modeling Strategies, 2nd edition](http://www.springer.com/us/book/9783319194240), page 97). Thoughtless use of stepwise regression, particularly when it is used to replace careful thinking about the subject matter, is certainly to be avoided. – EdM Dec 07 '17 at 17:45
  • Thanks all for your insights. This is certainly very helpful. – cgo Dec 08 '17 at 06:32
  • 1
    It is probably not harsh enough. Not only is thinking left out, more importantly we know that stepwise regression creates models that overfit the data, for which it is pretty spurious which variables are in the model and which usually predict poorly on a new sample. – Björn Dec 11 '17 at 10:15
1

Stepwise linear regression is a method wherein the features are selected one by one (if starting with no variables and slowly adding to the model) or removed (if starting with all the variables) so that the model is improved. It is a greedy algorithm where it finds the best solution at that particular step but does not find the over all best solution. I am fairly new to Data Science. Let me know if I'm wrong.

Steffi T
  • 11
  • 1