Should I use again Cross-Validation (retrain) on the amalgamation of the training and test set, fur tuning the regularization parameters?

Question

As far as I know, it is a good idea to retrain the model on all the data available (train, validation, and test) after finding the best Hyperparameters values by Cross-Validation.

However, some hyperparameters are sensitive to the dataset size, for example the regularization parameter.

According to that, should I retrain the model on the whole available data using the values of the hyperparameters I have found, or should I retrain the model on the amalgamation of the training and test set (this will be the new training set), and tuning the regularization parameter using the Cross-Validation and the validation set?

• The model that I'm using is XGBOOST classifier.

Answer 1

Yes, typically after obtaining a preferred model using cross-validation, you train the model on all data points — models which overfit will be discounted for due to poor accuracy. Thus, cross-validation is a way of assessing how the results of a model will generalize on an independent data set, rather than a way of obtaining parameters of a given model. Regularization with k-fold cross-validation, e.g., L1 or L2 is typically used for the latter. Let's say you pick some parameters for a particular model based on the accuracy of a particular fold without regularization, then depending on the distribution of those data points, your model could be overfitting (if the data points in the testing set are similar to that in training set) — thus in short, cross-validation is a way of assessing the performance obtained for a given model fixed by some parameters.