0

Will it be correct to use Mean absolute scaled error in non time series data? I've got a set which contains a lot of zeros, so errors like MAPE can not be used here. MASE based on difference between yt and yt-1 (formula like here), but I don't have time series, just observations of some activity of 200 people (independent), so maybe I can just use difference based on second and first person and so on? If not, do you know any percentage measures that I can use in that kind of set?

Skander H.
  • 10,602
  • 2
  • 33
  • 81
Roberto
  • 113
  • 3

1 Answers1

2

(The standard version of) MASE only works for time series. The purpose of the $Y_t$ - $Y_{t-1}$ term in the denominator is that it represents a naive forecast, that it is your best guess for $Y_t$ is $Y_{t-1}$.

In your case, your data is non-sequential and more importantly it is independent, so the idea of using $I_{n-1}$ as an estimator for $I_n$ doesn't really make sense (why $I_{n-1}$? Why not $I_{n-2}$ or $I_{n-199}$?)

To answer your 2nd question, it depends on what error you are trying to measures exactly ?

For dealing with the issue of zeros, a possible alternative for MAPE is sMAPE (Symmetric Mean Absolute Percentage Error). That might be a good option for you.


Addressing the comment:

  • When $Y_{emp}$ and $Y_{pred}$ are both equal to zero, than you set the sMAPE to 0 (since they are equal to each other, there is no error) (add an if clause to your calculation). Many software packages do that already, even with MAPE.
  • For the second problem, you can use the modified version of the sMAPE:

$$\text{sMAPE} = \frac{1}{N}\sum \frac{|y_{emp} -y_{pred}|}{{|y_{emp}|+|y_{pred}|}}$$

This gives you a 100% error in cases like you mentioned ($Y_{emp} = 0$ and $Y_{pred}=0.005$)

So the modified version of sMAPE could be:

$$\text{sMAPE} = \begin{cases} 0, & \text{ if } Y_{emp} = 0 & and & Y_{pred}=0 \\ \frac{1}{N}\sum \frac{|y_{emp} -y_{pred}|}{{|y_{emp}|+|y_{pred}|}} & \text{otherwise} \\ \end{cases} $$

If even 100% is too high a threshold, you can apply some sort of log transformation.


Since I first wrote this answer, I noticed elsewhere in this forum that Rob Hyndman (the original author of MASE) proposes a modified version of MASE for non-time series data. See here

If $e_j$ denotes a prediction error on the test data, then the scaled errors are $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{N}\sum_{i=1}^N |Y_i-\bar{Y}|}. $$ where $y_1,\dots,y_N$ denotes the training data.

Note here that $Y_{t-1}$ is replaced with $\bar{Y}$.

In the time series case, this would mean that the prediction is being compared to the predictions produced by a mean forecast instead of a naive forecast (as it is in the standard formulation of MASE).

Skander H.
  • 10,602
  • 2
  • 33
  • 81
  • I 've read about this measure but i can't use this because of situation when I have $Y_{emp}$ =0 and $Y_{pred}$ =0, then this measure is bad. Also When I check only freq, when $P_{emp}$ =0 and $P_{pred}$ =0,0005 then this measure going to its top (200%). I need error which can by comparable and don't have problems like I texted above. – Roberto Jan 09 '18 at 17:14
  • @Roberto see my edits. – Skander H. Jan 09 '18 at 17:45
  • Well that makes sens for probabilities. Thank You for your help. Also I am looking on the last formula in the wikipedia you mentioned and I think it solve a problem to both. Am I right? – Roberto Jan 09 '18 at 18:00
  • @Roberto the last formula in the wikipedia page - log(F/A) - requires strictly positive data and you said you have a lot of zeros? Also it would lead to cases where the error is negative for F < A . – Skander H. Jan 09 '18 at 18:13
  • I meant **SUM(abs[F-A]) / SUM(A+F)**. And yes, 80% of y in my set equal zero. – Roberto Jan 09 '18 at 18:16
  • @Roberto if your values are all positive than that formula is the almost the same as the second one (it is missing $\frac{1}{N}$ - but that might be a typo) - if you have negative values than it will have strange behavior for $F \rightarrow -A$ since it will give very large values and is undefined if $F=-A$. – Skander H. Jan 09 '18 at 18:24
  • I've got only non negative value (where 80% are zeros). The funny thing is that error between 1 and 1,1 is different (lower) than between 1 and 0,9. Is sMAPE comparable then? – Roberto Jan 09 '18 at 18:40
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/71417/discussion-between-alex-and-roberto). – Skander H. Jan 09 '18 at 18:45