Interpretation of Theil's U2 Statistic - "Forecasting Methods and Applications" book

Question

I was reading "Forecasting Methods and Applications" book and came across Theil's U statistic formula. I am facing difficulty in understanding the interpretation of Theil's U statistic.

According to my understanding, when U = 0 the forecasts are perfect. That means if U>0 then forecasts are not perfect. As we keep increasing U, the forecasts will keep becoming more and more imperfect. When it reaches 1, it means Forecast is equivalent to Naive Method. Hence, a general pattern can be seen that as U increases forecast is wrong and at 1 forecast is same as naive and hence by extrapolating this we can say that if U>1 then the forecast is worse than naive method forecast, i.e. APE of the forecast is even bigger than naive method forecasts.

I am able to understand the above interpretation but when I dig deep into the formula, I am not able to understand the below lines from the book.

For simplicity, I am also attaching equation (2.18) -

I am not able to understand 2 things here -

1. How do numerator and denominator represent the MAPE formulas for a general forecast and naive approach forecast? This is because MAPE of any forecast is represented by this formula: MAPE = Mean of (sum of (Forecast - Actual)/Actual). Hence, if we follow this formula of MAPE in the numerator, then Y(t) should be Y(t+1) i.e. (F(t+1) - Y(t+1)) / (Y(t+1) (mentioned above). This is because if we are taking the forecast of (t+1) then every number in the formula should be of (t+1). The same is for the denominator.

2. Why are we taking t from 1 to n-1? Shouldn't it be t from 1 to n?

The reason I am asking this question is that I want to know the underlying premise or logic behind using this formula.

Answer 1

jbowman already gave some very useful pointers. To follow up on your question:

If you were to intuitively explain the numerator and denominator, how would you do it?

I would call them both "nonstandard mean squared percentage errors", each period's error being expressed as a percentage of the previous observation. The numerator does this for the focal forecast, the denominator for the naive random walk one-step ahead forecast. If the ratio is less than one, then the nonstandard mean squared percentage error of the focal forecast is lower than the one for the naive method.

As such, this version of Theil's U (see below for my choice of words) is a type of "scaled" or "relative" forecast accuracy measure. It is similar to the MASE, which scales the focal forecasts's MAE by the MAE achieved (in-sample) by the naive random walk one-step forecast. "Relative" measures of accuracy divide a focal forecast's error measure by the one achieved by a benchmark method, and values less than one indicate that the focal forecast is better than the benchmark one.

A few things to keep in mind:

• This is pertinent:

  If you perform a Google search for Theil’s U, you’ll find an interesting assortment of different formulae, each purporting to be “Theil’s U”. According to Armstrong, the existence of multiple formulas “…has caused some confusion.”

  "This" Theil's U is a different thing than stuff I have seen elsewhere.

• Using the previous observation as the denominator in the percentage error is... strange.

• Theil's U only looks at one-step-ahead forecasts. That a method is worse than the naive method one step ahead does not necessarily mean it is worse multiple steps ahead. Compare Interpretation of mean absolute scaled error (MASE).

• The numerator and denominator are both means of (squared) percentage errors. As such, both are subject to What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? Most importantly, the MAPE (and therefore also the numerator and denominator in "this" Theil's U) are minimized by biased forecasts, so a biased forecast will more easily achieve $U<1$ than an unbiased one. Be aware of this.