2

I'm writing code to perform classification on novel data sets in our lab, and I'm confusing myself as to when I should be performing cross-validation. From this question I understand that I should choose hyper-parameters by cross-validation.

However, in the case that there are no hyper-parameters to tune, and all I want is to return labels for data points, do I need to cross-validate?

Community
  • 1
R Greg Stacey
  • 2,202
  • 2
  • 15
  • 30
  • Check http://stats.stackexchange.com/questions/1826/cross-validation-in-plain-english for start. – Tim Jul 12 '16 at 15:05
  • Thanks. From that I understand that I don't need to cross-validate if there are no parameters to tune. Is that right? – R Greg Stacey Jul 12 '16 at 15:22
  • What type of classification scheme are you using that has no hyper-parameters to tune? Is it one that might nevertheless be prone to over-fitting? You might then find need some type of validation, although bootstrapping might be more useful than cross-validation. Numbers of cases, predictors, and class representation (particularly the number of cases in the smaller classes) would help here. – EdM Jul 12 '16 at 15:32
  • I'm using a Naive Bayes on 2 classes and 10 features, with about 20,000 positive cases and 500 negative. The question about what to do if there are no hyperparameters is hypothetical, since I am performing cross-validation for hyperparameters like data whitening and class balancing. However I'm confused if I need to do any kind of cross-validation after that, or whether I can just classify. I don't think I'm interested in e.g. estimating the model performance - I just want labels for each data point. – R Greg Stacey Jul 12 '16 at 15:41

1 Answers1

3

Ideally, always cross-validate because getting the results on different splits allows you to better estimate your model performance. In practice, you do it if you estimate that you don't have enough data to train/test your model properly.

This is not directly linked to hyper parameters. e.g. a valid experimental protocol is to split your data in training/validation and test sets. You train your model on the training set You choose your hyper-parameters that perform the best on the validation set. When all done, you evaluate your final model on the test set.

paul_gay
  • 46
  • 1
  • Thanks. If I'm not interested in estimating model performance, and I just want to attach classes to data points, could I skip cross-validation? Or is it good practice to always cross-validate? – R Greg Stacey Jul 12 '16 at 15:46
  • If you just want to attach labels and don't care how accurate those labels are, why even build a model? Just assign labels arbitrarily. If you care how good those labels are then use something like cross validation or an independent test set to measure performance. – dsaxton Jul 12 '16 at 23:00
  • Yes, cross-validation is only for evaluation. When in production, you want to use all your training data to learn the best possible model. – paul_gay Mar 01 '18 at 11:56