4

I am a newbie in the Machine Learning world, I completed the course (very good by the way) of Andrew Ng on Coursera. This question is very software-independent. I would like to know, when you draw a learning curve, do you represent the training error and CV error (using the metric that we want like rmse or $R^2$ for linear regression) as a function of the training set size? Or do you represent instead training error and test error as a function of the training set size? I have seen lot of people plotting the learning curve for the test error, whereas in the course of Andrew Ng I have seen the learning curve for the CV error.

I attach as an example some curve that I got few months ago using Python.

Thanks a lot for the clarification, best regards

example of learning curve

Sven Hohenstein
  • 6,285
  • 25
  • 30
  • 39
Jean-Philippe Fontaine
  • 41
  • 2

1 Answers1

1

It represents training error and testing error. No cross validation involved (usually we have one big fixed testing data set, and changing the size of training samples to produce the curve).

My answers here gives you more details:

How to know if a learning curve from SVM model suffers from bias or variance?

Haitao Du
  • 32,885
  • 17
  • 118
  • 213
  • But if we are in the search for the best model can't we plot learning curves comparing training and CV error from many models to check which looks more robust? – Felipe Cruz Jul 13 '17 at 14:44
  • 3
    Learning curves are not meant to help you choose between competing models, they are meant to tell you if more training data would be useful (I think, I don't find them very useful in practice). – Matthew Drury Jul 13 '17 at 15:39
  • @FelipeCruz check Matthew's comments. – Haitao Du Jul 13 '17 at 16:06
  • 2
    @MatthewDrury I feel it is a useful tool in practice. I have seen to many people tuning parameters or collect more data without guidance. – Haitao Du Jul 13 '17 at 16:09
  • I suppose i've never been in the situation that I could just collect more data, so maybe that is on me. On the other hand, they are not much use in tuning parameters, much more useful is the model complexity vs performance plots. – Matthew Drury Jul 13 '17 at 16:15
  • @MatthewDrury thanks. I found this very clear explanation confirming what you just said: http://scikit-learn.org/stable/modules/learning_curve.html#learning-curve – Felipe Cruz Jul 13 '17 at 16:29
  • @MatthewDrury my work involves evaluating some really different types of models, such as linear or 100 layer deep network. Using this can help me to see which way to go instead of just trying many things... – Haitao Du Jul 13 '17 at 18:05