2

I am doing a forecast using robust exponential smoothing methods and to determine / measure the forecast accuracy I want to use robust measurements as well. As I am not really familiar with robust measures I would like you to ask what good robust accuracy measures exist?

I have found MADM/median here on stackexchange (see below the quote) which seems to be appropriate in my case. For the calculation of the MADM I would use for xi the error difference between real data and predicted. x would be the vector of all errors. However, I could not detect any academic discourse on the proposed MADM/median.

You may want to look into the median absolute deviation from the median (MADM). That is: MADM=median(|xi−median(x)|) I suspect a better nonparametric analogy to the coefficient of variation would be MADM/median, rather than IQR/median.

(cf. here the answer of gung at Oct 5 '12 at 14:41: A robust (non-parametric) measure like Coefficient of Variation -- IQR/median, or alternative?)

Community
  • 1
Tunc Jamgocyan
  • 103
  • 1
  • 5
  • I'm not sure what you really mean by "academic discourse," but you can find plenty of information about MAD, such as its [Wikipedia article](http://en.wikipedia.org/wiki/Median_absolute_deviation). If by "MADM/median" you mean the ratio of these two statistics, then I cannot see what its purpose might be for assessing forecast accuracy. It only is a (crude) measure of shape of a distribution of *strictly positive* numbers. – whuber Feb 14 '15 at 15:27
  • Thanks for your help again @whuber. What good robust accuracy measures would you propose to use? – Tunc Jamgocyan Feb 14 '15 at 15:51
  • I would select accuracy measures that are appropriate for the application, the data, and the risk averseness of my client. Because I know none of those things in your circumstance, I cannot in good conscience recommend any particular measure for you. – whuber Feb 14 '15 at 16:51
  • I am predicting a univariate time series. The times are modeled as numbers, starting from t_1 = 1, t_2 = 2, ... , t_n = n which means that the we have n months and thus n observations. The prediction method I use is the Holt's linear trend method and the Robust exponential smoothing method of Cipra (1992): Robust exponential smoothing, Journal of Forecasting Volume 11, Issue 1, pages 57–69, January 1992. As I use a robust forecast technique, I would like to use appropriate robust accuracy measures as well. – Tunc Jamgocyan Feb 14 '15 at 17:10

1 Answers1

1

Firstly, it is important to decide what is the loss function you want to use in your evaluation. In their recent review of alternative error measures, (Davydenko and Fildes, 2016) recommend the use of the symmetric linear loss because of the following reasons:

Fitting a statistical model usually delivers forecasts optimal under quadratic loss. This, e.g., happens when we fit a linear regression. If our density forecast from statistical modelling is symmetric, then forecasts optimal under quadratic loss are also optimal under linear loss. But, if we stabilise the variance by log-transformations and then transform back forecasts by exponentiation, we get forecasts optimal only under linear loss. If we use another loss, we must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function (see examples of doing this in Goodwin, 2000).

Let’s assume we want to empirically compare two methods and find out which method is better in terms of a symmetric linear loss (since this type of loss is commonly used in modelling). If we have only one time series, it seems natural to use a mean absolute error (MAE). Also, MAE is attractive as it is simple to understand and calculate (Hyndman, 2006).

Then they notice that

Potentially, MAE has the following limitation: absolute errors follow a highly skewed distribution with aheavy right tail, which means that MAE is not robust (in other words, it is a highly inefficient estimate).

(Davydenko and Fildes, 2016) propose the following solution to define a robust MAE measure:

If distribution of absolute errors is heavily skewed, MAE becomes a very inefficient estimate of the expected value of absolute error. One simple method to improve the efficiency of the estimates while not introducing substantial bias is to use asymmetric trimming algorithms, such as those described by (Alkhazeleh and Razali, 2010).

References:

Davydenko, A., & Fildes, R. (2016).Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions.John Wiley & Sons Inc.

The full text is available here.

Turbofly
  • 46
  • 2