Reducing the number of variables in a multiple regression

Question

I have a large data set consisting of the values of several hundred financial variables that could be used in a multiple regression to predict the behavior of an index fund over time. I would like to reduce the number of variables to ten or so while still retaining as much predictive power as possible. Added: The reduced set of variables needs to be a subset of the original variable set in order to preserve the economic meaning of the original variables. Thus, for example, I shouldn't end up with linear combinations or aggregates of the original variables.

Some (probably naive) thoughts on how to do this:

1. Perform a simple linear regression with each variable and choose the ten with the largest $R^2$ values. Of course, there's no guarantee that the ten best individual variables combined would be the best group of ten.
2. Perform a principal components analysis and try to find the ten original variables with the largest associations with the first few principal axes.

I don't think I can perform a hierarchical regression because the variables aren't really nested. Trying all possible combinations of ten variables is computationally infeasible because there are too many combinations.

Is there a standard approach to tackle this problem of reducing the number of variables in a multiple regression?

It seems like this would be a sufficiently common problem that there would be a standard approach.

A very helpful answer would be one that not only mentions a standard method but also gives an overview of how and why it works. Alternatively, if there is no one standard approach but rather multiple ones with different strengths and weaknesses, a very helpful answer would be one that discusses their pros and cons.

whuber's comment below indicates that the request in the last paragraph is too broad. Instead, I would accept as a good answer a list of the major approaches, perhaps with a very brief description of each. Once I have the terms I can dig up the details on each myself.

Answer 1

Method 1 doesn't work. Method 2 has hope depending on how you do it. It's better to enter principal components in descending order of variance explained. A more interpretable approach is to do variable clustering, then reducing each cluster to a single score (not using Y), then fit a model with the cluster scores.

Answer 2

You might consider using a method like LASSO that regularizes least squares by selecting a solution that minimizes the one norm of the vector of parameters. It turns out that this has the effect in practice of minimizing the number of nonzero entries in the parameter vector. Although LASSO is popular in some statistical circles many other related methods have been considered in the world of compressive sensing.

Answer 3

In chapter 5 of Data Mining with R, the author shows some ways to choose the most useful predictors. (In the context of bioinformatics, where each sample row has 12,000+ columns!)

He first uses some filters based on statistical distribution. For instance, if you have half a dozen predictors all with a similar mean and s.d. then you can get away with just keeping one of them.

He then shows how to use a random forest to find which ones are most useful predictors. Here is a self-contained abstract example. You can see I've got 5 good predictors, 5 bad ones. The code shows how to just keep the best 3.

set.seed(99)

d=data.frame(
      y=c(1:20),
      x1=log(c(1:20)),
      x2=sample(1:100, 20),
      x3=c(1:20)*c(11:30),
      x4=runif(20),
      x5=-c(1:20),
      x6=rnorm(20),
      x7=c(1:20),
      x8=rnorm(20,mean=100, sd=20),
      x9=jitter(c(1:20)),
      x10=jitter(rep(3.14, 20))
      )

library(randomForest)
rf=randomForest(y ~ ., d, importance=TRUE)
print(importance(rf))
#         %IncMSE IncNodePurity
# x1  12.19922383    130.094641
# x2  -1.90923082      6.455262
# ...

i=importance(rf)
best3=rownames(i)[order(i[, "%IncMSE"], decreasing=TRUE)[1:3]]
print(best3)
#[1] "x1" "x5" "x9"

reduced_dataset=d[, c(best3, 'y')]

The author's last approach is using a hierarchical clustering algorithm to cluster similar predictors into, say, 30 groups. If you want 30 diverse predictors you then choose one from each of those 30 groups, randomly.

Here is some code, using the same sample data as above, to choose 3 of the 10 columns:

library(Hmisc)
d_without_answer=d[,names(d)!='y']
vc=varclus(as.matrix(d_without_answer))
print(cutree(vc$hclust,3))
# x1  x2  x3  x4  x5  x6  x7  x8  x9 x10 
#  1   2   1   3   1   1   1   2   1   3 

My sample data does not suit this approach at all, because I have 5 good predictors and 5 that are just noise. If all 10 predictors were slightly correlated with y, and had a good chance of being even better when used together (which is quite possible in the financial domain), then this may be a good approach.

Answer 4

This problem is usually called Subset Selection and there are quite a few different approaches. See Google Scholar for an overview over related articles.