4

I am forecasting a weekly commodity price series. I use a rolling window for estimating my model, and from each window I make point forecasts for one and two steps ahead.

I want to investigate forecast optimality. Diebold (2015, p. 334, list item d) indicates that one of the desirable properties of a good forecast is

Optimal forecasts have $h$-step-ahead errors with variances that are nondecreasing in $h$ <...>.

I would like to test this for my forecasts. I have two series of forecasts:

  • a series of 1-step-ahead forecasts
  • a series of 2-step-ahead forecasts

with their corresponding series of forecast errors:

  • a series of 1-step-ahead forecast errors (a vector $e_1$)
  • a series of 2-step-ahead forecast errors (a vector $e_2$)

Question 1: How may I test whether the variance of the 2-step-ahead forecast errors is greater than or equal to the variance of the 1-step-ahead forecast errors?

My thoughts

The relevant null hypothesis will be

$$\text{H}_0: \text{Var}(\varepsilon_1) \leqslant \text{Var}(\varepsilon_2),$$

where $\varepsilon_i$ is the population counterpart of $e_i$, for $i=1,2$. Or should I consider

$$\text{H}_0: \text{Var}(\varepsilon_1) > \text{Var}(\varepsilon_2)$$

instead? This is a subject-matter issue. From a technical perspective, I am aware that strict inequality in the null hypothesis is problematic.

Testing $\text{H}_0: \text{Var}(\varepsilon_1) \leqslant \text{Var}(\varepsilon_2)$ would be easy if I could rely on some extra assumptions such as

  • the observations in each of the series are $i.i.d.$

If I had this assumption, I could apply some standard test for equality of variances, e.g. Levene's test or Brown–Forsythe test.

However, the 2-step-ahead forecasts may well follow an MA(1) process (Diebold, 2015, p. 334, list item c), or probably have some more complicated dependence structure; recall that my forecasts may be suboptimal in many ways -- it is a real-world exercise after all.

Then I could perhaps approximate my forecast errors by ARMA processes, estimate their unconditional variances and compare them.

Question 2: How would I formally test that the unconditional variance from one ARMA process is greater than or equal to that of another ARMA process?

If my thinking above is correct, I may post Question 2 separately.

References

  • Diebold "Forecasting in Economics, Business, Finance and Beyond" (online textbook; available here; version of 14 December 2015)

Related newer question: "How to test that a sequence of variances rank ascendingly?".

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • I may be wrong but it seems to me that testing $H_0$ is different from checking if variances are non-decreasing in $h$. The latter seems to be more of a mathematical property than a statistical one. I'm really interested in question 2 tough. – Regis A. Ely Jan 29 '16 at 16:38
  • @RegisA.Ely, could you explain how the two are different? I observe two sample quantities and I want to test whether their population counterparts satisfy an inequality. It looks pretty straightforward to me, but I might be missing something. – Richard Hardy Jan 29 '16 at 19:00
  • Oh, I think it's my mistake. So Diebold is saying that the **population** variances of the h-step-ahead forecast errors should be non-decreasing in h if the forecast is optimal? I'm not sure how you are defining $e$ and $\epsilon$. – Regis A. Ely Jan 29 '16 at 19:51
  • @RegisA.Ely, well, I don't know that myself. I am used to employing sample data to make inference about population, but perhaps this is not the case. Perhaps it is enough to just look at the sample values and see if they satisfy the inequality. Interesting... By the way, the way I introduced $\varepsilon$ may not be natural; my point was to show that I am testing a hypothesis about a population property rather than a sample property. But as I indicated, I am not sure this makes sense. – Richard Hardy Jan 29 '16 at 20:15
  • The forecast error would be $e_t = Y_t - \hat{Y_t}$, but I am not sure what you mean by the population forecast error. That is why I said property *d* looks more like a mathematical property than a statistical one, but I'm not sure. Maybe someone can clarify that. – Regis A. Ely Jan 29 '16 at 20:28
  • @RegisA.Ely, perhaps you are right that comparing the estimated error variances is enough; Diebold sort of suggests that on p. 336, if I am reading him correctly (your comments helped me get another perspective of how to read p. 336). – Richard Hardy Jan 29 '16 at 20:32
  • Yes, exactly. He just says to examine if the sample h-step-ahead forecast error variances are decreasing. – Regis A. Ely Jan 29 '16 at 20:40
  • A quick note to myself: consider applying the Diebold-Mariano test with a one-sided alternative hypothesis. – Richard Hardy Aug 05 '16 at 12:32

1 Answers1

0

Hopefully the mean of your errors is zero. If you are willing to assume that for this purpose, then you can compare the distributions of the square errors, using the Wilcoxon test.

chrishmorris
  • 820
  • 5
  • 5