2
mean(abs(pred - true)) / mean(abs(true))

I defined above metric that works well for measuring reconstruction error for zero-mean signal data (1D); does it have a name? It's not MAPE.

OverLordGoldDragon
  • 654
  • 4
  • 11

1 Answers1

2

It's very closely related to MAD/Mean, where you just take the mean of the series for the denominator, without taking absolute values. If your series is nonnegative, the two notions coincide.

MAD/Mean is a weighted MAPE, with the actuals as the weighting (Kolassa & Schütz, 2007, Foresight). Your MAD/MAV analogously is a weighted MAPE with the absolute values of the actuals as the weighting.

Whether or not this KPI "works well"... well, if your series is stationary, then it is equivalent to the plain MAD, which is minimized by the median. So minimizing this KPI will incentivize you to report the predictive median. If your series is symmetric, then this will be the same as the mean. If not, your predictions will be biased. That's fine if you want it, but it should be kept in mind (Kolassa, 2020, IJF). The discussion at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? is related.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thanks for the info. The sole motivation is _scale-invariance_; reconstructing a signal with twice the amplitude gives twice the error, even though "relative" error is the same, and I can't just take mean of non-absolute signal as that's ~0. I also asked [here](https://dsp.stackexchange.com/q/71243/50076); if there's really no name for it, perhaps time to coin one. – OverLordGoldDragon Nov 04 '20 at 16:16
  • Hmm, important problem in this metric: not robust to _sparsity_, or simply signal _duration_. E.g. if we simply add zeros to an existing signal, and assume the prediction signal maps them perfectly, then MAE/MAV will begin to _increase_. I've settled for the RMS (i.e. L2 norm) instead. – OverLordGoldDragon Nov 05 '20 at 00:48
  • Yes, that is precisely the fact that MAE is minimized by the median. If you add probability mass at zero, then the median will tend towards zero. In extreme cases, when half or more of your mass is zero, the flat zero prediction is MAE-optimal. This used to confuse people who do intermittent demand forecasting (e.g., sales of rarely sold items at supermarkets). – Stephan Kolassa Nov 05 '20 at 07:18
  • Indeed, though I was in fact referring to the _normalization_ (i.e. denominator), and it turns out I haven't really solved the [problem](https://stats.stackexchange.com/q/495242/239063). – OverLordGoldDragon Nov 05 '20 at 18:50