1

If you cannot acquire a workable test data-set to test a model, you can use cross validation as an alternative to validate the model.

However, I'm unsure of the true end purpose of cross validation...

Is CV useful for simply generating a performance metric by providing a way to test your model? Or is it a means to creating a final model?

Put differently: do I use my "best" model from whichever given fold of the CV process (i.e., the "best" model using some partition of the data) as my final model? Or do I perform CV, report my performance metric, and then re-create the model using ALL of my available data to create a final model?

Community
  • 1
theforestecologist
  • 1,777
  • 3
  • 21
  • 40

2 Answers2

2

Is CV useful for simply generating a performance metric by providing a way to test your model? Or is it a means to creating a final model?

For both. It generates performance metrics to test your model (perhaps, on different parameters).

It is also a means of creating the final model, because you would "often" use cross-validation to choose the best parameters of the model. You then use the best parameters on all of the data to generate the final model. However, while reporting please report the cross-validation error.

Community
  • 1
discipulus
  • 726
  • 4
  • 14
  • What exactly is "cross-validation error"? How is it calculated? – theforestecologist Oct 08 '16 at 04:30
  • 1
    You train a model on training data. Use that model to predict on the test data. The performance of the model on the test data is called cross-validation error. – discipulus Oct 08 '16 at 04:31
  • Ah. So it's just relative to what it's reporting performance of (i.e., the model's ability to predict values for the test data). It could theoretically consist of MSE, RMSE, AUC, MAE, etc. , then? – theforestecologist Oct 08 '16 at 04:34
  • 1
    Yes! Metric is your choice and depends on the task at hand and also on the data. Cross-validation error is used more generically; it may be accuracy or AUC or accuracy or similar in clasification and MAE or MSE or similar in regression. – discipulus Oct 08 '16 at 04:37
  • Error and accuracy are not the same; they are the flip sides of the same coin. – rolando2 Jan 28 '17 at 23:28
0

From Zhang 1997:

The purpose of data splitting is to choose the best model or to study model stability and predictive performance. The final estimation of the model parameters should come from the entire data set (Myers, 1990; Shao, 1993).

Myers RH. 1990. Classical and modern regression with applications. 2nd Edn. Boston: PWS-KENT Publishing Company.

Shao J. 1993. Linear model selection by cross-validation. Journal of American Statistical Association 88: 486±494.

ZHANG, LIANJUN. 1997. Cross-validation of non-linear growth functions for modelling tree height–diameter relationships. Annals of Botany 79.3: 251-257.

theforestecologist
  • 1,777
  • 3
  • 21
  • 40