Importance of variables

Question

In a set of data, I have one dependent variable and 50 independent variables. Out of these 50, how can I find the variables which are important in estimating the dependent variable?

Answer 1

A good approach that actually eliminates variables is Lasso Regression. Basically, you will be splitting up your dataset into several "folds" (usually 5 or 10), where you will fit a regression model to one of them and then test its accuracy against the remaining folds (then repeat for each other fold). This is called cross validation, which attempts to correct for the overfitting bias inherent when you use the same data to fit and test your model.

They key with Lasso is that you are minimizing not just the MSE (as in normal regression), but adding in a "regularization" penalty as well. In this case, it takes the form:

$$\gamma \sum |a_i|$$

Where the $a_i$ are your regression coefficients and $\gamma$ is a non-negative "tuning parameter".

You will be doing two separate optimizations while selecting your variables:

1. Pick a value for $\gamma$, then calculate the cross-validation error for the resulting model. (This is the average MSE across the folds not used to fit the model).
2. Adjust $\gamma$ and repeat

This will produce a curve of Cross Validation Error vs $\gamma$. We will want to find the $\gamma$ that gives you the smallest cross validation error. Let's call that $\gamma^*$.

Note that $\gamma^*$ is still a sample statistic (i.e., estimate), the model with the lowest $\gamma$ is not necessarily the best at predicting within your allowed space of models.Therefore, it is recommended that we err on the simpler side for a model, given the uncertainty about which model actually gives the lowest cross validation error.

In the case of Lasso, that means that you should choose $\gamma \geq \gamma^*$ such that the cross validation error ($CVE$) is equal to $CVE_{\gamma^*}+s(CVE_{\gamma^*})$ (i.e., find the gamma that gives you a model whose cross validation error is one standard deviation higher than the cross validation error you got at the empirical "optimum" gamma of $\gamma^*$ (the sample standard deviation for CVE is calculated from the folds generated by $\gamma^*$)