2

My sample size is 550, and I build a regression model with 80 features. In particular, I am using LASSO and ridge regression with 5-fold leave one-out cross-validation. To evaluate the models, I am using correlation (between the predicted and actual value) and mean absolute error (MAE). The correlation I obtain is around .5, and the MAE is around .8. The value of the variable that the model predicts is in the range of 0-8.

My results seem to be promising (but not perfect). Since the sample size is small and the feature set is large, I wonder if it is reasonable to run CV on whole data and evaluate the performance of the model based on this? I am working on an academic publication, and I wonder if this is an acceptable method (with limitations reported). Or, should I use bootstrapping to resample the data, and apply CV on only the training set? I appreciate any insights.

renakre
  • 755
  • 1
  • 9
  • 25
  • 2
    Please add some more details about what you are trying to accomplish with this regression model. This would probably be substantially overfit if you have a standard linear regression with that many samples and features. What aspect of "the performance of the model" are you trying to evaluate? Bootstrapping doesn't "generate more data"; rather, it re-uses your data in a way that mimics re-sampling from the underlying population. It can be, however, a more efficient use of your data than CV for evaluating your model-building process. – EdM Feb 20 '17 at 21:43
  • @EdM thanks for the feedback. Do my updates make sense? I appreciate any help since this is a bit urgent! – renakre Feb 20 '17 at 21:50

1 Answers1

2

It seems that you have performed ridge regression and LASSO, choosing the penalty parameter values by 5 repetitions of leave-one-out CV, with either the correlation between observed and predicted or the mean absolute error (MAE) as your evaluation measure. Your data set actually isn't that small relative to the number of features as you are using penalized regressions, but you can do a bit better in your model building and evaluation.

First, you should consider whether either the correlation coefficient or MAE are the best measures for choosing model parameters. I think that many would prefer to use the mean-squared error, as that is what both ridge and LASSO start with as their errors before adding the penalization term on the coefficients.

Second, once you have chosen your measure for fit, to choose your penalty parameter values you might be better off doing true 10-fold cross validation to find the best values. Leave-one-out CV can be somewhat noisy, and 5 repetitions might not be enough to overcome that.

Third, what you can perhaps best accomplish is to validate the model building process by repeating your entire process (including your cross-validation to choose your penalty parameter values) over multiple bootstrap samples of the original data, and examining your evaluation measure, over those multiple models, on your original data set. That gives you (and your audience) a good estimate of how well your process would have behaved if you had the chance to take multiple samples from the population. Note that for LASSO you will almost certainly select different features in each of the bootstraps; you might find it informative to look at that issue directly.

This thread and this thread may also be useful. ISLR (Chapters 5 and 6), ESLII (esp. Chapter 7) and Harrell's rms course notes provide more details and examples.

Community
  • 1
EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thank you very much for the detailed response, I really appreciate it! With my current model and approach, I wonder if one can argue that there is over-fitting ? – renakre Feb 20 '17 at 22:40
  • Rather than "argue that there is over-fitting," test your model-building process on multiple bootstrap samples and see how it performs. – EdM Feb 20 '17 at 22:43
  • Sir, using the `resample` method of sklearn and 10-fold cv, I obtain consistently the same results on different samples of my data. Would that mean I can be more confident about the prediction model and the results? – renakre Feb 21 '17 at 08:35
  • 1
    Technically you can be more confident about the model-building process, as you can't completely rule out some irregularity in your particular data sample that might affect this particular model. Summarizing the performance of multiple models from bootstraps applied to the original data set can document the optimism in the main prediction model, as well as showing the variability in regression coefficients and selected features. See for example Chapter 5 of the rms course notes linked in my answer (click on Handouts on the linked page). – EdM Feb 21 '17 at 12:49