1

I have some data, which I split into training, cross-validation and test sets. I built two models, that I trained on the training set and optimised using the cross-validation set (e.g. finding the optimal polynomial degree and regularisation parameter). I then combined the training and the cross-validation sets into one and used this larger set test to train both algorithms and calculated the test error. However, my dataset is quite small (530 entries), and the test error varies quite a bit, so the range of $R^2$ seems to be 0.40-0.65 for both. My two questions are:

  1. Is it valid to combine the training and cross-validation sets after the algorithm has been optimised?
  2. Is it a good idea to calculate the test error multiple times (e.g. 100) for both algorithms (following random selection of the training, cross-validation and test sets) and look at the average to compare the two models?

Sorry if the question is primitive, I'm very new to Machine Learning!

GingerBadger
  • 405
  • 3
  • 7

1 Answers1

2

Firstly, there is no need to split the data set into a training, CV and testing set. It is usually sufficient to do a train and test split and using the training set to perform CV.

When doing 10-fold CV, your training set is split into 10 buckets, hence if your dataset has 530 observations, 85% of this results in 450 observations in your training set. When performing 10-Fold CV, your validation set will have 10% of 450, 45 data points for calculating accuracy/error and 405 data points for training your model.

Because you have relatively few data points, I assume your algorithm is running fast and time is not an issue. Instead of performing K-fold cross validation I would perform leave one out CV. In this method, you train your model using $n-1$ (449) data points and test against 1. The errors are then averaged.

When performing CV don't look at only the best hyper parameters which reduce the errors most, but also pay particular attention to overfitting. A simple model which significantly reduces errors is sufficient, as opposed to higher degree models which reduce errors only marginally.

To answer your first question, yes. After performing any type of CV, you always train your final model against your FULL training set.

Your second question, it is okay to calculate testing error multiple times, but be careful what you do with the data. The reason we performed CV in the first place, was to estimate test error. We train the model first and test it against data it has never seen before. That is the whole reason behind CV. Therefore, CV is a good representation of testing error.

If we start to repeatedly calculate the testing error and tweaking our model, we immediately start overfitting the testing data. Do not do this. It is okay to collect data on how the two models perform after you have optimized your CV error and just report these results.

I like to do bar plots of the two model error and report the data. The average on its own does not convey the whole story, hence a bar/violin plot will how the first, second quartile, outliers and the mean. It makes a good story.

Sada93
  • 390
  • 1
  • 11
  • Thanks so much! Just a few follow-up questions to make sure I understand it all correctly: 1. I used CV to optimise my model (e.g. the regularisation parameter λ). Do you suggest that I do leave-one-out CV 450 times for each value of λ and use mean errors to select the best λ? 2. Having picked the best model, which error do I then report -- the average cross validation error I got from CV, or the error from the test set (80 entries) based on the model trained on 450 entries (train + CV)? – GingerBadger Aug 09 '17 at 10:29
  • 1
    Sorry, one more question: what do you think about [this post](https://stats.stackexchange.com/questions/61783/variance-and-bias-in-cross-validation-why-does-leave-one-out-cv-have-higher-var) claiming that leave-one-out cross-validation has a large variance and is, therefore, inferior to 10-fold cross-validation? – GingerBadger Aug 09 '17 at 13:55
  • 1
    1. Yes, but after reading that post I may change my mind about LOOCV. 2. You finally report the error on your test set based on the complete model trained. Thank you for that post, learned a lot. – Sada93 Aug 09 '17 at 16:01
  • Sorry, I'm still confused. Why is it not valid to report the average cross-validation error, if it is even supposed to be slightly larger than the test error? Moreover, re that post, why does a large variance matter when we're looking at the mean? – GingerBadger Aug 09 '17 at 16:55
  • 1
    It's not that you don't report explicitly report CV error, you show it in your report, but the final error you present will be the testing error from your hold out set. Again, the mean of errors alone doesn't convey all the information, variance shows the spread in your error estimate. Like I said, bar plots help convey this information. A longer/thicker bar plot means the variance is higher. – Sada93 Aug 09 '17 at 21:51