24

I have problem with defining the unit of accuracy in a regression task.

In classification tasks is easy to calculate sensitivity or specificity of classifier because output is always binary {correct classification, incorrect classification}. So I can count good/bad answers and based on the confusion matrix calculate some measurements.

But in regression tasks the output is a number. So I can't just say is it correct/incorrect -- instead I should measure "how far from true solution am I".

So what be the unit of accuracy in regression task?

Firebug
  • 15,262
  • 5
  • 60
  • 127
Mr Jedi
  • 373
  • 1
  • 2
  • 7
  • 4
    Have you heard of coefficient of determination, a.k.a. $R^2$? That's a popular measure. – Richard Hardy Mar 22 '15 at 13:31
  • 3
    Have you looked here: http://en.wikipedia.org/wiki/Forecasting#Forecasting_accuracy ? – 404pio Mar 22 '15 at 15:16
  • 1
    If you're looking at the relative "goodness" of two or more models, you could also look into [AIC](http://en.wikipedia.org/wiki/Akaike_information_criterion) or [BIC](http://en.wikipedia.org/wiki/Bayesian_information_criterion) – Chris C Mar 22 '15 at 17:02
  • 1
    @RichardHardy the $R^2$ can be useful but should definitely not be blindly suggested. I think the answer to the question depends on what is important in the regression. A often used approach in regression is to minimize the MSE. A downside of the $R^2$ is that when we add another factor into our regression the $R^2$ cannot go down even if the variable essentially adds no information. – Jan Oct 18 '18 at 12:42
  • 1
    @Jan, good points. I suggested to the OP looking at $R^2$ simply because it is the standard even though it is neither always relevant nor flawless. Good that you added some points of warning. It is also true that not all criticism of $R^2$ hits the point. For example, you may or may not worry about the fact that $R^2$ can only grow by adding more variables. It depends on what you are looking for. – Richard Hardy Oct 18 '18 at 13:14
  • check this for normalized eucledian similarity as a candidate measure: https://stats.stackexchange.com/questions/136232/definition-of-normalized-euclidean-distance – Charlie Parker Mar 03 '21 at 19:54
  • Check this for normalized euclidean similarity NES/1-NED as a candidate measure: https://stats.stackexchange.com/questions/136232/definition-of-normalized-euclidean-distance the question and answers already contain great details. – Charlie Parker Mar 03 '21 at 19:55
  • I think you are looking for Mean Absolute Percentage Error (MAPE). Or maybe. Here is the formula: [MAPE](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error). If looking for some kind of accuracy measure, then use 1-MAPE. – sam Mar 13 '21 at 20:09

1 Answers1

25

You should ask yourself what were you trying to achieve with your modeling approach.

As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for classification, we only get into specifics when we run into dichotomization, usually in more CS oriented machine learning, such as trees or SVMs).

So, let's measure it, shall we? If $x_i$ is the truth and $\hat x_i$ your model output, for sample $i$, here's the error:

$$\epsilon_i = x_i - \hat x_i$$

You could measure the mean error $\sum_i \epsilon_i$, but it turns out that, doing that, positive and negative errors cancel, giving you no way to know how good your model actually performs!

So, what people do in general, is to use these measures:

  • Squared error:

    $$\text{SE}=\sum_i^n \epsilon_i^2$$

  • Mean squared error:

    $$\text{MSE}=1/n \times \text{SE}$$

  • Root mean squared error:

    $$\text{RMSE}=\sqrt{\text{MSE}}$$

  • Relative mean squared error (do not confuse this for the RMSE, root mean squared error):

    $$\text{rMSE}={n-1\over n}{{\sum_i^n \epsilon_i^2}\over {\sum_i^n (x_i - \mathbb E(x))^2}}= {\text{MSE} \over Var(x)}$$

  • $\text{R}^2$:

    $$\text{R}^2=1 - \text{rMSE}$$

  • Absolute error:

    $$\text{AE}=\sum_i^n \sqrt{\epsilon_i^2}=\sum_i^n |{\epsilon_i}|$$

  • Mean absolute error:

    $$\text{MAE}=1/n \times \text{AE}$$

And many, many others. You can find them around on the site (see for example How to interpret error measures?).

Firebug
  • 15,262
  • 5
  • 60
  • 127