1

I have built a forecasting model for a company. Since it is dedicated to practical usage, I prefer to use the relative error parameter (like MAPE, SMAPE, & MASE) as a measurement for my model performance and display it on the dashboard (the percentage error is easier to understand by user).

I found an interesting reference about the criteria of a good forecast result based on MAPE from Lewis (1982): enter image description here

...but in my case, I can't use MAPE for some dataset because there is some zero actual demand so it can't be used as denominator in MAPE. So, I use SMAPE & MASE instead, but I can't find any good references about good forecast result criteria based on SMAPE & MASE, nothing like above.

So, I'd really appreciate any help/suggestion about it :), Thank You!

Reference: Lewis, C. D., 1982. Industrial and Business Forecasting Methods. London: Butterworths.

Defa Ihsan
  • 19
  • 4
  • A somewhat related question that talks about interpreting the size of MASE: ["Interpretation of mean absolute scaled error (MASE)"](https://stats.stackexchange.com/questions/124365). – Richard Hardy Jun 24 '20 at 10:31

2 Answers2

5

Those Lewis numbers are fairly arbitrary, you cant just say that a 20% error is good forecasting because some guy wrote it in a book 40 years ago.

The acceptable margin or error completely depends on the problem domain. In some situations a model that gives a 20% error will be great, in others it will be unusable. I know its tempting to rely on general rules like the ones you posted because they feel 'objective', but they are ultimately arbitrary and cant override common sense and domain expertise.

James
  • 506
  • 1
  • 5
4

+1 to Gordon's answer.

Forecast accuracy "guidance" or "benchmarks" are not worth the bits they take up. They are typically derived from surveys on a convenience sample. I went into some detail in a critique of such benchmarks in an article (Kolassa, 2008, Foresight: The International Journal of Applied Forecasting). Yes, it's a couple of years old, but every single thing I pointed out is still valid, and will be forever.

Sometimes you have series on which a MAPE of 10% is great, because the series is very stable. In other cases, you may be deliriously happy to reach 70%. There is just no way to say in general.

Better: compare your forecast to a very simple internal benchmark, i.e., a very simple forecasting algorithm like the overall mean or the random walk. If you can improve on that, great. (If not, don't be surprised.) Consider analyzing the value of your improved forecast in monetary terms, if you can simulate the decisions made on the basis of the forecast.

Note that the MASE already does something in this vein, in that it compares your forecast MAE to the MAE achieved in-sample by a random walk forecast for $h=1$. So if your MASE is below 1, you have improved on that. But note that even a MASE above 1 can be good, especially for longer-range forecasts. And of course, you could use a variant of the MASE, where you use a different benchmark forecast for the denominator.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • I agree with this. Processes vary in complexity and stability and there is no gold standard here that is true across processes. I would add that a good error rate is what is acceptable to the people you work for :) – user54285 Jun 24 '20 at 20:19