1

In linear regression, the coefficient of determination $R^2$ is a normalized measure for prediction accuracy. In machine learning, performance measures are not computed by estimating the performance on the same data that has been used for training, because this would yield a too optimistically biased estimator.

I wonder why in linear regression $R^2$ is estimated from the predictions on the training data that has been used to estimate the model parameters. I would expect this to be optimistically biased, especially in cases of a low sample size / number of parameters ratio, and situations of overfitting will thus go unnoticed.

Why is it not estimated with leave-one-out (aka "n-fold cross-validation") or bootstrap?

cdalitz
  • 2,844
  • 6
  • 13
  • 1
    This is more a question about the history and culture of machine learning vs. statistics, I think. There's a few related questions on the site about this. – mkt Jul 05 '21 at 17:41
  • 1
    e.g. https://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning – mkt Jul 05 '21 at 17:42
  • It is as biased as other in-sample statistics. Depending on the application, it definitively makes sense to calculate out-of-sample or CV R-squared. I do it quite often. – Michael M Jul 05 '21 at 19:32
  • @mkt Do you mean that a possible peference of a biased estimator is is due to "culture"? I guess that there are other features of the in sample estimator of $R^2$ that make it attractive over a less biased extimator. And is the issue of quality measure estimation discussed in the linked thread? – cdalitz Jul 05 '21 at 21:03
  • 1
    Yes, I think traditional stats classes teach that these methods are valid because of theoretical work. The fact that the assumptions of that theory are frequently unmet doesn't really end up being emphasised, and so solutions like CV R2 end up being ignored. – mkt Jul 06 '21 at 06:16

0 Answers0