Methods of variable selection

Question

A study stated that it used forward selection to chose variables for a multivariable regression model (in this case logistic) to evaluate association between predictor and outcome. They started with theoretically plausible variables until they had a decent model, and then tested additional covariates by adding them to the model and checking if the predictor effect size (in this case odds ratio) was changed by 5%. They did not include covariates that did not change the predictor effect size by this (I presume arbitrary?) threshold. I had read the faults of stepwise selection, which uses P-values, but had not come across this method before.

1. Is this a recognised method and is it any more/less problematic than conventional forward stepwise?

2. Can someone give an undergrad-level list of common variable selection techniques to read up on? Specifically for epidemiology/sociology, rather than machine learning. I'm coming across penalised regression (lasso/ridge) and PCA (for modelling propensity scores).

3. I assume variable selection methods depends on the goal of the model (eg. prediction, inference, propensity score, etc)? Can you tell me how/if each technique is suited to each common goals?

Answer 1

This sounds like a bad idea, though I can't exactly tell what the criterion for adding a predictor to the model was. How can a "predictor's odds ratio" be "changed" by 5% if it was just added to the model?

5% indeed was pulled out of you-know-where. The techniques Spatzle mentioned, forward/backward/best subset, are an improvement, because they base the variables chosen on a metric like AIC, which balances how well the model fits the data with the number of variables in the model (in an attempt to avoid overfitting). However, these algorithms too are actually usually a bad idea! They are "greedy," so they often fail to optimize well, and more importantly they still tend to overfit. It's also hard to interpret p-values/significance from these models since the variables are adaptively selected.

LASSO is a good place to start with variable selection. Ridge regression is less directly suited to variable selection. PCA does dimensionality reduction; variable selection is a specific type of dimensionality reduction that maintains variable definitions. PCA usually will spit out weird linear combinations of the original variables that will be hard to interpret.

The fact is, in the sciences where model interpretability and inference may be more important than predictiveness (i.e. you don't want to just throw all the data in a Random Forest or Neural Net), nothing really beats a careful, scientific approach to the problem. Make sure the variable codings are clean and sensible, and deal with missing data appropriately. Then choose covariates, a model, and functional forms for the covariates based on your scientific knowledge, perhaps tweaking the model slightly after fitting to improve model diagnostics. Some of your coefficients won't be significant, which is fine, and better yet your p-values will be honest!

Answer 2

I'd stick my neck and say that this described method sounds like abusive use of p-values. Moreover, some variables can be insignificant as individuals but significant as a group (see explanation regarding Linear Regression here)

Generally speaking, there are three common methods for variable selection:

a. Best subset

b. Forward selection

c. Backward selection

You can read some more here and here.

If you know Linear Regression well, you must have encountered $R^2$ and its wonderfully evil twin $\bar{R}^2$. The latter can be used as a criterion for adding/subtracting a covariate, like many others (AIC, Mallows' $C_p$ and more, see the above links).

As we cannot use these methods with logistic regression, my personal favorite criteria is proportional confusion matrix (which is confusion matrix divided by the number of samples). If there's an error type you can define as severer, you should look at minimizing it (usually the case with biostatisics, when true negative is worse than false positive); Otherwise, look at maximizing the correct prediction error (e.g modelling Obama vs. Romney, there should be no difference of severity between type 1 and type 2 errors). This is my favorite criterion, while working with backward selection. This can change, of course, depending on the number of predictors and the type of problem.