Is LOOCV valid for OLS within timeseries framework

Question

I have a OLS model under the timeseries framework. Apart from several independent variables I have one which is a lagged version of the Y variable.

Is LOOCV on this model valid?

It seems to be not ok but is there any justification as in references for (or against) performing LOOCV on this type of model?

In time series order matters so it does not make sense to train on the future and test on the past. Here is a nice blog that speaks on approaches for validation of a time series model. https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/ — Jon, Feb 15 '18 at 19:38

score 4 · Answer 1 · answered Feb 16 '18 at 01:28

4

It depends on what model you are using. If you can write the model as a (nonlinear) autoregression, then yes, LOOCV is ok. It sounds like your model is of the form

$$ y_t = f(y_{t-1}, x_t, z_t, \varepsilon_t)$$

where $x_t$ and $z_t$ are exogenous variables and $\varepsilon_t$ is a white noise error process. That would fit within the models discussed in Bergmeir, Hyndman & Koo (2018), where we prove that under some conditions CV can work for time series models.

answered Feb 16 '18 at 01:28

Rob Hyndman

51,928
23
126
178

Surely my name, plus "we prove" is pretty clear. – Rob Hyndman Apr 24 '18 at 23:02
Oops, it escaped me haha, I was actually thinking about your comment under my answer actually when I wrote that, where you missed it. No big deal though. – Firebug Apr 25 '18 at 13:49

Firebug · Answer 2 · 2018-02-15T19:39:45.653

2

Is LOOCV on this model valid?

Naively? Totally not.

You'd be using data from the future to assess the past. Your performance estimate would be completely wrong.

edited Feb 15 '18 at 19:39

answered Feb 15 '18 at 19:19

Firebug

15,262
5
60
127

1

Not necessarily. See https://robjhyndman.com/publications/cv-time-series/ – Rob Hyndman Feb 16 '18 at 01:29

score 0 · Answer 3 · answered Feb 17 '18 at 18:18

Time series models are statistical models where data is an ordered sequence of values of a variable at equally spaced time intervals. (See: NIST/SEMATECH e-Handbook of Statistical Methods )

LOOVC or k-fold CV are not appropriate for time series as they do not opperate within the constraints of the definition of time series models. In LOOVC or k-fold CV, you would shuffle your data (shuffle past values with future values) and split the data into partitions. Each partition would loose order of sequence and thus lose significance as values occurring over time.

Simply, LOOCV and k-fold CV are appropriate for models that assume data is identically and independently distributed (iid).

An alternative to LOOCV and k-fold CV are time series CV methods. After a quick search, I believe this is post describes the methodology

Choosing inner cross validation strategy for modeling time series data

Also, you can see that depending on the tools you use for modeling, there may already be time series specific CV tools. See the following example:

sklearn.model_selection.TimeSeriesSplit

Note that unlike standard cross-validation methods, successive training sets are supersets of those that come before them.

Is LOOCV valid for OLS within timeseries framework

3 Answers3