Normalized RMSE

Question

I have several time-series in a VAR(1) and, due to some of them haven't the same unit of measure, I'd like to estimate the RMSE in percentage. I know that it could be done in several ways (see below) but I don't know precisely which is the one that fits better a forecast evaluation problem. I hope you could help me.

Examples of normalized RMSE: $$ RMSE_1 = \sqrt{\frac{1}{n}\sum_i\left(\frac{Y_{forecast_i}-Y_i}{Y_i}\right)^2} \\ RMSE_2 = \sqrt{\frac{1}{n}\sum_i\left(\frac{Y_{forecast_i}-Y_i}{Y_{forecast_i}}\right)^2} \\ RMSE_3 = \frac{\sqrt{\frac{1}{n}\sum_i\left(Y_{forecast_i}-Y_i\right)^2}}{mean(Y)} $$

score 6 · Accepted Answer · answered Mar 23 '16 at 09:08

6

You have also other choices that are commonly used in such cases, e.g. relative absolute error

$$ \text{RAE} = \frac{ \sum^N_{i=1} | \hat{\theta}_i - \theta_i | } { \sum^N_{i=1} | \overline{\theta} - \theta_i | } $$

root relative squared error

$$ \text{RRSE} = \sqrt{ \frac{ \sum^N_{i=1} \left( \hat{\theta}_i - \theta_i \right)^2 } { \sum^N_{i=1} \left( \overline{\theta} - \theta_i \right)^2 }} $$

mean absolute percentage error

$$ \text{MAPE} = \frac{1}{N} \sum^N_{i=1} \left| \frac{\theta_i - \hat{\theta}_i}{\theta_i} \right| $$

where $\theta$ is true value, $\hat \theta$ is the forecast and $\overline{\theta}$ is a mean of $\theta$ (see also https://www.otexts.org/fpp/2/5).

answered Mar 23 '16 at 09:08

Tim

108,699
20
212
390

when implementing RAE or RRSE, can you recommend on a reasonable way to avoid `inf`? I thought about using `mean` rather than `sum` and max the denominator with an `epsilon` value – ihadanny Sep 15 '19 at 15:48
@ihadanny why do you see inf values? – Tim Sep 15 '19 at 17:11
when $\theta$ are constant the denominator is 0 – ihadanny Sep 16 '19 at 19:09
@ihadanny then the error measure does not make sense. – Tim Sep 16 '19 at 19:22

score 2 · Answer 2 · answered Mar 02 '15 at 15:30

A possible way would be to normalize the RMSE with the standard deviation of $Y$:

$NRMSE = \frac{RMSE}{\sigma(Y)}$

If this value is larger than 1, you'd obtain a better model by simply generating a random time series of the same mean and standard deviation as $Y$.

Normalized RMSE

2 Answers2

Linked