0

I recently studied K-fold cross validation, but I didn't get why this result even matter, since it does not help to improve the generalization ability of our model and the model will work the same on new data?

Ajay Singh
  • 149
  • 1
  • 9
  • I'm not entirely clear what you are asking about. The point of CV lies in applying it to *different* models and *choosing* the model that has the lowest CV error. Could you clarify? – Stephan Kolassa Jul 05 '16 at 12:26
  • you are talking about validation, please see this link http://www.csie.ntu.edu.tw/~b92109/course/Machine%20Learning/Cross-Validation.pdf k-fold cross validation is used to predict the performance of the estimator, and it generally gives a better estimate, but i'm asking why the estimate % value matter, whether I'm giving an estimate without k-fold or with k-fold my model remains the same and it will give the same results on new data no matter what my estimates are, So why people use this? – Ajay Singh Jul 05 '16 at 12:38

3 Answers3

3

Cross validation has two uses:

  1. assessing a model's likely performance on new data
  2. comparing multiple models' performance

In use #2, you apply CV to different models, assess each one's likely out-of-sample performance per #1, and then choose the better one.

So, assuming we already have chosen a model, what's the remaining point in #1? After all, CV won't improve the performance of the model chosen.

The point now is that a prediction is good... but what is almost as important is a measure of how good that prediction is likely to be. Having an idea of the reliability of your prediction helps your subsequent processes. If you have a very reliable prediction, you need to invest less resources in mitigation efforts for mispredictions than if you had a less reliable prediction.

For instance, my own field is forecasting daily sales in supermarkets per stock-keeping unit (SKU), for replenishment. The point forecast is pretty much useless, because we want to achieve a certain service level. To stock enough product, we need a quantile forecast, which (given a distributional assumption) is conceptually similar to a point forecast plus a notion of the variability of the forecast error. We don't use time series cross validation for this, but it would certainly be a valid approach.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Actually i read it in a book "To finish our evaluation process, we will introduce new method cross-validation. We usually partition our dataset into a training set and a testing set. However, partitioning the data, results such that there are fewer instances to train on, and also, depending on the particular partition we make (usually made randomly), we can get either better or worse results. Cross-validation allows us to avoid this particular case, reducing result variance and producing a more realistic score for our models. The usual steps for k-fold cross-validation are the following:" – Ajay Singh Jul 05 '16 at 13:22
  • 1. Partition the dataset into k different subsets. 2. Create k different models by training on k-1 subsets and testing on the remaining subset. 3. Measure the performance on each of the k models and take the average measure. – Ajay Singh Jul 05 '16 at 13:22
  • just to clearify, CV won't tell the accuracy of prediction? – Ajay Singh Jul 05 '16 at 13:31
  • It does. More precisely, it *attempts* to estimate the accuracy of your model on a new dataset. – Stephan Kolassa Jul 05 '16 at 13:32
  • so what i get is it just gives a better estimate of the accuracy of prediction? – Ajay Singh Jul 05 '16 at 13:35
  • 1
    Exactly. (Apart from using CV to *select* a model based on this information.) And I argue that this is often useful. – Stephan Kolassa Jul 05 '16 at 13:42
3

You should avoid point estimates as much as possible. What you really want (at a minumim) is an estimate of centrality and an estimate of variability, or (even better) a distribution of measurements of "accuracy"[1]. CV gives you a distribution, so it's better than a one-time (point) estimate.

When people talk about CV for selecting between models, they should clarify that it's nested CV they're talking about: an outer loop of Cross-validation that includes the entire process for creating, training, and scoring each competing model. Within that outer loop, there would be a CV loop for each model. (This inner CV is what you get when you use a modeling function and set something like nfolds=10, or cv=TRUE or whatever, to use CV for that individual model.)

The point being that models look perfect when you imagine them, and if you don't rigorously probe them you will end up deploying a model that works far less well than you've promised. Or one that is no better than random, or perhaps even worse.

[1] Accuracy is often not what you actually care about. I put "accuracy" in quotes because it may actually be Precision or some other measure that you actually care about in a particular situation.

Wayne
  • 19,981
  • 4
  • 50
  • 99
1

In a nutshell: cross validation gives you information about how your model(s) will perform on data not seen during training. So, yes, using CV will technically not change the model you obtain from training using your training data - but, the information from CV will help you selecting one "best suited" model out of multiple models trained on it (e.g. different model types and hyper parameters, which, in contrast to CV itself, can influence how your model will fit the data). This means that without using CV, you will likely have worse real world performance in the end, just because of selecting a suboptimal model type or parametrization.

BTW: there are a couple of answers around CV that will give you more details, if you want to read upon them:

Community
  • 1
geekoverdose
  • 3,691
  • 2
  • 14
  • 27