2

Just a question on regression model evaluation statistics. Here we go.

I seem to be under the impression that $R^2$, MSE, and RMSE are all very closely related and essentially all play a part in determining the fit of a model, but somehow I am still confused about which one, then, I should be using as a final determinant of how good/bad my model is performing, or how is it that I can consider all three as separate indicators (with them being so similar)?

It just seems like no matter what, the model with the $R^2$ closest to 1 also has the lowest MSE and RMSE values. I read that "sometimes" the $R^2$ doesn't determine the fit properly, and so the other metrics should be the determinants of fit, but if they all correlate (one goes up, the other goes up, and vice versa), is there even reason to separately consider them? (<- my bad if this seems redundant)

That's all I've got. Thanks for your time!

mkt
  • 11,770
  • 9
  • 51
  • 125
Angela S.
  • 61
  • 1
  • 6
  • 1
    You might be interested in [this thread](https://stats.stackexchange.com/questions/250730/what-is-the-mathematical-relationship-between-r2-and-mse) and [this one](https://stats.stackexchange.com/questions/32596/what-is-the-difference-between-coefficient-of-determination-and-mean-squared) – mkt Sep 28 '19 at 18:58
  • RMSE is like "average magnitude of error". While you can average the absolute values of the regression errors, "absolute value" was not used historically as it does not have a continuous derivative. So the errors were squared to make them positive, the mean of these squares was found, and then the square root of that mean was calculated. As for R-squared, it tells you what fraction of the dependent data variance is explained by the model - it is exact for a straight line and approximate for other models. Sometimes the modeling goal is to find a model with the smallest peak-to-peak error. – James Phillips Sep 28 '19 at 21:07

1 Answers1

2

Mean square error (MSE) and root mean square error (RMSE) obviously have a one-to-one relationship, as MSE can't be negative. Confusion can arise, however, from what MSE means in a particular context. As the Wikipedia page on mean squared error notes:

In regression analysis, the term mean squared error is sometimes used to refer to the unbiased estimate of error variance: the residual sum of squares divided by the number of degrees of freedom.

If that is what you have in mind for MSE then the standard $R^2$ reported for linear regression necessarily increases as MSE decreases, other things being equal, as it is defined as:

$$ R^2 \equiv 1 - {SS_{\rm res}\over SS_{\rm tot}}$$

where $SS_{\rm res}$ is the residual sum of squares and $SS_{\rm tot}$ is the total sum of squares.

Sometimes, however, MSE refers to the mean-squared error in predictions made on a new data set. It's quite possible for a regression model to overfit the data at hand by including too many predictors relative to the number of observations. In that case you could have very small MSE from your regression but large MSE when you apply the regression model to a new data set. The adjusted $R^2$ often reported by statistical software starts to take overfitting into account by adjusting for the numbers of predictors and observations.

One example of using MSE in this second context is to separate the data into a training set and a test set, develop the regression on the training set, and then evaluate MSE on the test set as a test of the quality of your regression model. That approach works best if you have very large numbers of observations. With smaller data sets, analogous approaches based on cross-validation or bootstrapping use MSE of predictions to serve a similar purpose.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • If I got this correctly, I tried using just the training data to run my linear regression model, then scored it to get my measures (R^2, MSE, etc), then I ran it separately on a model using training data as the input to be learned and the test data as the predictor input (to be predicted, ie new data(?). I got pretty different evaluation metrics such that the solo training data set had a higher rate of acuracy (R^2), lower MAE, MSE, and RMSE than the model using the test data. Granted I actually did what you explained in your example properly, now what does this indicate about my model? – Angela S. Sep 28 '19 at 23:48
  • 1
    @AngelaS. what you describe is fairly standard; MSE on predictions for test data will typically be lower than MSE in the training set used to build the regression model. The question is: how much lower? If you have thousands of cases, the MSE on held-out test data is a good measure of model quality. If you have fewer cases you risk losing power with a separate held-out test set; see chapter 5 of [ISLR](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20First%20Printing.pdf) for an introduction to cross validation and bootstrapping for evaluating a model. – EdM Sep 29 '19 at 15:57
  • Thank you for both providing me with a practical explanation to my query as well as providing some reading material for me to reference further. You've been so helpful :) – Angela S. Oct 02 '19 at 19:44