Multiple Regression and number of parameters to include for a learning algorithm

Question

I am quite new to Machine Learning and come from a computing background.

1. I have a quite big set of features (~50) with about 4k observations. Is it correct thinking to include all of them in a multiple linear regression model? Wouldn't that possibly introduce overfitting? If that's the case, is it safe to manually identify which of the features would matter the most for the model and for future predictions, and include only those?

2. Given prior experience with Java, would you recommend any other tools than Java for Machine Learning computations? I am aware of some Java libraries such as Weka, Mahout and other languages such as Python, MATLAB, and R (which I hear is quite slow).

Answer 1

I suggest fetching the free PDF book An Introduction to Statistical Learning with Applications in R and heading to chapters 3 through 9 which cover the introductory scope of the supervised learning problem and show you how to learn from your data. You didn't say anything else about your data -- how many samples, how many responses, whether it's a classification problem -- but I would suggest 50 is not in fact a huge set of predictors. The methods described in the ISLR book show you how to find the most effective models from the predictors and responses given your training and testing data. Insofar as R goes, you'll find it is quite fast and can do in two or three lines what requires dozens of lines of other languages. R is your friend for this sort of exploration.

Answer 2

How much your model over-fits depends on, among other things, how many observations you have. As a rule of thumb, for observational studies, it gets bad when you have fewer than 10–20 per coefficient estimate (excluding the intercept). But you can cross-validate the full model & see, rather than reduce it just in case. There are two (not mutually exclusive) approaches that help when over-fitting is a problem:

1. Data reduction. Here's where your expert comes in. It's not just a matter of looking at the names of the candidate predictors: consider the variability or prevalence of each in your sample, which are likely to be measuring much the same thing, which are measured most accurately, which have fewer missing values, &c. And selecting may need to be supplemented, or even supplanted, by combining: from simple averages or differences to principal components of variable clusters. At this stage you're also considering for which ones it's worth allowing for non-linear relationships to the response, or for interactions. Section 4.7.7 here gives a way to use the deviance of the full model to guess how much data reduction will be helpful.

2. Regularization. The idea of this is to shrink the coefficient estimates to correct for the optimism introduced by overfitting. Ridge regression shrinks estimates for all coefficients towards zero; LASSO shrinks some to zero, thereby performing variable selection; the elastic net combines both procedures. How much to shrink can be guided by cross-validation or a modified version of Akaike's Information Criterion.

Of course model selection is a big topic. The two books I've found most helpful are these:

Harrell (2002), Regression Modelling Strategies

Hastie et al. (2009), Elements of Statistical Learning

Answer 3

Including ~50 features in a linear regression could result in multicollinearity; this will happen in at least two of your features are highly correlated. In extreme case, if at least two features are perfectly correlated, your software package will not compute the results because the feature matrix is singular. Multicollinearity will cause your coefficients estimates to be off.

If you indeed have multicollinearity problem, you could use principal component analysis to find principal components that explain most most variation of your data, and use them as your features. Principal components are not correlated by construction.