1

If I have used MASE to calculate non time-series data error (as described by Dr. Rob Hyndman here), how can I know if the score received is good or not? Since it is not a time-series, a random walk naive model is irrelevant here, and the threshold of 1 which is usually used doesn't seems to be a good comparison. Is it correct? if yes - what should be the consideration to decide what score is good or not? if not - why?

BTW it seems that even for time series data, a score of 1 and bellow is not always an achievable target as describe here - so how can you know what is considered to be sufficient?

YonGU
  • 11
  • 1
  • 7

1 Answers1

3

Based on his textbook for a non-seasonal time series, a useful way to define a scaled error uses naïve forecasts: $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{T-1}\sum_{i=1}^T |y_t-y_{t-1}|} $$

Acoording to his answer here, it is possible to scale the data using the mean as the base forecast. If $e_j$ denotes a prediction error on the test data, then the scaled errors $q_j$ are: $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{N}\sum_{i=1}^N |Y_i-\bar{Y}|} $$ where $y_1,\dots,y_N$ denotes the training data.

Note here that $Y_{t-1}$ is replaced with $\bar{Y}$

The MASE compares "your" forecast against a naive benchmark forecast calculated in-sample due to the MASE denominator is calculated in-sample, not in the holdout sample. The MASE is a metric for comparing errors to a user-defined baseline:

$$ MASE= mean(|q_{j}|) $$

For interpretation of MSE which is not an easy job please have a look at this earlier thread on interpreting the MASE might be helpful from @Stephan Kolassa

My recommendation:

  • Check this answer to not confused about the calculation of MSAE and MAE
  • if you have a single series, then the MASE is less informative than the MAE, since the MASE is simply the MAE scaled by a factor that does not depend on the forecast. The MASE makes sense once you have multiple series on different levels, where you can't very well compare "raw" MAEs.

Note: the scaling statistic $Q$:

$$ \text{MASE}=\text{MAE}/Q $$

Mario
  • 341
  • 2
  • 12