23

In a question I asked recently, I was told that it was a big "no-no" to extrapolate with loess. But, in Nate Silver's most recent article on FiveThirtyEight.com he discussed using loess for making election predictions.

He was discussing the specifics of aggressive versus conservative forecasts with loess but I am curious as to the validity of making future predictions with loess?

I am also interested in this discussion and what other alternatives there are that might have similar benefits to loess.

Community
  • 1
a.powell
  • 1,047
  • 7
  • 19
  • If your x-variable is time, it would be dangerous to use loess to predict into the future (which would be outside the range of the data). But that doesn't mean you can't use loess to make predictions more generally. – Glen_b Jul 28 '16 at 17:46
  • @Glen_b out of curiosity what would something I could "more generally" predict? – a.powell Jul 28 '16 at 17:50
  • 7
    Imagine a nonlinear relationship between the the proportion of people inclined to vote for party A and the unemployment rate (along with other predictors -- effects for the individual states for example). Further imagine there's new unemployment figures just become available; within the range of values experienced in the training set, but not necessarily a value represented in that set (e.g. past unemployment is between 5 and 12% and we now have a figure of 8.3%, forecast to be steady). Then we could use loess to predict the proportion voting A, without going outside 5-12% unemployment. – Glen_b Jul 28 '16 at 18:07
  • 1
    @Glen_b Thank you. That is a wonderful illustration of how it can be used for forecasts. – a.powell Jul 28 '16 at 18:18

1 Answers1

29

The problem with lowess or loess is that it uses a polynomial interpolation. It is well known in forecasting that polynomials have erratic behavior in the tails. When interpolating, piecewise 3rd degree polynomials provide excellent and flexible modeling of trends whereas extrapolating beyond the range of observed data, they explode. Had you observed later data in the time series, you definitely would have needed to include another breakpoint in the splines to obtain good fit.

Forecasting models, though, are well explored elsewhere in the literature. Filtering process like the Kalman filter and the particle filter provide excellent forecasts. Basically, a good forecast model will be anything based on Markov chains where time is not treated as a parameter in the model, but previous model state(s) are used to inform forecasts.

AdamO
  • 52,330
  • 5
  • 104
  • 209