1

I will perform time-series prediction and I will report the accuracy of my system with a measure like RMSE or MAE.

However, the variables I will predict are in different ranges. So let's say one is in millions (~ 1e6) whereas the other is a fraction, so ~ 1e-1. So when I report the MAE on these two variables, one will be 10m times bigger than the other one, while my system's accuracy is similar on the two.

So what are good ways to obtain a comparable performance measure?

The ways I can think of is :

  1. Dividing the score with the mean of the data.
  2. Expressing each residual as a fraction of the true value, so if the true value is x, and my prediction is Xp, I can measure the error as |x-Xp|/x

Can you guide me into a meaningful way to solve this problem?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
jeff
  • 1,102
  • 3
  • 12
  • 24
  • What about scaling (e.g. center and reduce) the values to predict to have comparable errors ? – RUser4512 Sep 14 '15 at 08:17
  • @RUser4512 thanks. Yes one approach I have in mind (and probably apply) is min-max normalizing the data between -1 and 1. In this case the errors should be comparable, but is there a way for un-normalized data? – jeff Sep 14 '15 at 14:13

1 Answers1

1

You could look into normalized and relative error measures.

Specifically, I recommend the root-relative squares error (RRSE) as a good analogue to root mean squared error (RMSE). As @Tim pointed in his answer:

$$ \text{RRSE} = \sqrt{ \frac{ \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 } { \sum^N_{i=1} \left( \overline{\theta} - \theta_i \right)^2 }} $$ [...] where $\theta$ is true value, $\hat \theta$ is the forecast and $\overline{\theta}$ is a mean of $\theta$ (see also https://www.otexts.org/fpp/2/5)

These relative error measures can be easily compared across different dependent variable ranges. Basically, if $\text{RRSE} = 1$ you aren't doing better than simply outputting the mean of the observations for all of them, which can be interpreted as a no-information baseline model.

Community
  • 1
Firebug
  • 15,262
  • 5
  • 60
  • 127