13

I am fitting a stepwise logistic regression on a set of data in SPSS. In the procedure, I am fitting my model to a random subset that is approx. 60% of the total sample, which is about 330 cases.

What I find interesting is that every time I re-sample my data, I am getting different variables popping in and out in the final model. A few predictors are always present in the final model, but others pop in and out depending on the sample.

My question is this. What is the best way to handle this? I was hoping to see the convergence of predictor variables, but that isn't the case. Some models make much more intuitive sense from an operational view (and would be easier to explain to the decision makers), and others fit the data slightly better.

In short, since variables are shuffling around, how would you recommend dealing with my situation?

Many thanks in advance.

Btibert3
  • 1,154
  • 1
  • 13
  • 23

5 Answers5

16

If you're going to use a stepwise procedure, don't resample. Create one random subsample once and for all. Perform your analysis on it. Validate the results against the held-out data. It's likely most of the "significant" variables will turn out not to be significant.

(Edit 12/2015: You can indeed go beyond such a simple approach by resampling, repeating the stepwise procedure, and re-validating: this will lead you into a form of cross-validation. But in such a case more sophisticated methods of variable selection, such as ridge regression, the Lasso, and the Elastic Net are likely preferable to stepwise regression.)

Focus on the variables that make sense, not those that fit the data a little better. If you have more than a handful of variables for 330 records, you're at great risk of overfitting in the first place. Consider using fairly severe entering and leaving criteria for the stepwise regression. Base it on AIC or $C_p$ instead of thresholds for $F$ tests or $t$ tests.

(I presume you have already carried out the analysis and exploration to identify appropriate re-expressions of the independent variables, that you have identified likely interactions, and that you have established that there really is an approximately linear relationship between the logit of the dependent variable and the regressors. If not, do this essential preliminary work and only then return to the stepwise regression.)

Be cautious about following generic advice like I just gave, by the way :-). Your approach should depend on the purpose of the analysis (prediction? extrapolation? scientific understanding? decision making?) as well as the nature of the data, the number of variables, etc.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • 2
    +1 for highlighting the importance of model interpretation. I won't add anything about the uninformed ML approach (or ensemble methods) with more complex cross-validation schemes, because I feel you already said what's really matter here: (1) feature selection through resampling is hardly interpretable in isolation (i.e., by comparing one result after the other), and (2) it all depends if we are seeking for a predictive or an explanatory model. – chl Dec 10 '10 at 21:03
  • Thanks for your insight. I have done some pre-screening to narrow my search space and simply want to find the best model for prediction with the fewest variables. I am only throwing 7 predictors into the model, which as I understand it, should be ok. I understand the idea of sticking with a sample, but on the flip side, my model was fundamentally different and shows that the results are entirely sample-dependent, which made me pause. – Btibert3 Dec 10 '10 at 21:30
  • @Btibert3 Right: when the results vary among random subsets of your data, you can take that as evidence that the independent variables are not strong or consistent predictors of the independent variable. – whuber Dec 10 '10 at 22:03
12

An important question is "why do why do you want a model with as few variables a possible?". If you want to have as few variables as possible to minimize the cost of data collection for the operational use of your model, then the answers given by whuber and mbq are an excellent start.

If predictive performance is what is really important, then you are probably better off not doing any feature selection at all and use regularized logistic regression instead (c.f. ridge regression). In fact if predictive performance was what was of primary importance, I would use bagged regularized logistic regression as a sort of "belt-and-braces" strategy for avoiding over-fitting a small dataset. Millar in his book on subset selection in regression gives pretty much that advice in the appendix, and I have found it to be excellent advice for problems with lots of features and not very many observations.

If understanding the data is important, then there is no need for the model used to understand the data to be the same one used to make predictions. In that case, I would resample the data many times and look at the patterns of selected variables across samples to find which variables were informative (as mbq suggests, if feature selection is unstable, a single sample won't give the full picture), but I would still used the bagged regularized logistic regression model ensemble for predictions.

Dikran Marsupial
  • 46,962
  • 5
  • 121
  • 178
  • 1
    +1 for the pointer to regularized logistic regression. It's unclear how one could formally "look at patterns" when resampling the "data many times," though. That sounds a lot like data snooping and therefore seems likely to lead to frustration and error. – whuber Dec 11 '10 at 19:07
  • 5
    Feature selection when the selection is unstable will always be a recipe for frustration and error. Using only one sample cuts down on the frustration, but increases the likelihood of error as it encourages you to draw inferences about the relevant features for the problem based on what works best on the particular sample you look at - which is a form of over-fitting. The re-sampling gives you an idea of the uncertainty in the feature selection - which is often just as important. In this case we shouldn't draw any strong conclusions about the relevant features as there isn't enough data. – Dikran Marsupial Dec 11 '10 at 20:42
  • Good point; I hate when people only count mean from resampling, it's such a waste. –  Dec 11 '10 at 23:38
10

In general, there are two problems of feature selection:

  • minimal optimal, where you seek for smallest set of variables that give you the smallest error
  • all relevant, where you seek for all variables relevant in a problem

The convergence of predictor selection is in a domain of the all relevant problem, which is hell hard and thus requires much more powerful tools than logistic regression, heavy computations and a very careful treatment.

But it seems you are doing the first problem, so you shouldn't worry about this. I can generally second whuber's answer, but I disagree with the claim that you should drop resampling -- here it won't be a method to stabilize feature selection, but nevertheless it will be a simulation for estimating performance of a coupled feature selection + training, so will give you an insight in confidence of your accuracy.

  • +1 I worry that lots of resampling will only be confusing and misleading. Resampling in a controlled way, via cross validation or a hold-out sample for verification, obviously is not problematic. – whuber Dec 11 '10 at 19:08
6

You might glance at the paper Stability Selection by Meinshausen and Buhlmann in J R Statist. Soc B (2010) 72 Part 4, and the discussion after it. They consider what happens when you repeatedly divide your set of data points at random into two halves and look for features in each half. By assuming that what you see in a one half is independent of what you see in the matching other half you can prove bounds on the expected number of falsely selected variables.

chl
  • 50,972
  • 18
  • 205
  • 364
mcdowella
  • 403
  • 4
  • 2
2

Don't use stepwise! See my paper

Peter Flom
  • 94,055
  • 35
  • 143
  • 276