4

I am working with some time series data and I developed a linear regression model to make future predictions. The model has the following form:

$$ y\left( t \right) =\sum _{ i=1 }^{ M }{ { \alpha }_{ i }{ f }_{ i }\left( x\left( t \right) \right) } $$

where ${ f }_{ i }\left( x\left( t \right) \right) $ is a nonlinear function and ${ \alpha }_{ i }$ are the parameters learned during training.

The data has 120 observations from which 108 are used for training. In order to determine the appropriate number of terms $M$, a time series cross-validation was used on the 12 remaining observations, as described by Prof. Hyndman on his website. I applied a slightly modified version so I describe it below:

  1. Start with one model term.
  2. Estimate the parameter coefficients of the model using the first 108 observations, forecast the remaining 12 and compute the errors.
  3. Move the first observation of the test set to the training set and repeat step 2, i.e. train on 109 observations, forecast on the remaining 11 observations, and compute the errors.
  4. Step 3 is repeated for the remaining observations in the test set, i.e. until train on 119 observations, forecast the last observation available, and compute the last error.
  5. All the errors are combined into a Mean Absolute Error value. This represents the CV error for the model.
  6. Add a new model term, and repeat steps 2-5.
  7. The procedure is repeated until a specified number of terms is reached.

Such technique produces the following plot:

Cross-validation error plot

From the plot, the final model selected has 47 terms because this produced the lowest cross-validation error. However, there are several local minima, the most notorious at 2, 17 and 32 terms.

My questions are:

  1. Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?
  2. Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?
  3. Why not use the first or the second local minimum as the best number of terms?

Any help is greatly appreciated!

jroberayalas
  • 577
  • 1
  • 5
  • 19
  • "cross-validation was used on the 12 remaining observations" -- this is not clear, did you use 10-fold cross-validation (ten times leaving out 12 observations our of 120) or simply computed the error on the fixed test set consisting of 12 observations? – amoeba Dec 15 '15 at 16:21
  • @amoeba I computed the error on the fixed test set of 12 observations, but using the approach described [here](http://robjhyndman.com/hyndsight/tscvexample/) – jroberayalas Dec 15 '15 at 16:25
  • Then do 10-fold cross-validation instead. This will essentially give you 10 error curves instead of only 1 and you can average them. This will decrease the noise and make it smoother. – amoeba Dec 15 '15 at 16:27
  • @amoeba I considered using a k-fold CV, but it seems it is not the best approach to do given that this is a time-series problem. That is the reason why I used the method I shared in the [link](http://robjhyndman.com/hyndsight/tscvexample/). It seems that this approach is usually used in order to save the dependencies within the data. This was briefly discussed on another post [here](http://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection) which basically refers to the original [source](https://www.otexts.org/fpp/2/5) by Hyndman. – jroberayalas Dec 15 '15 at 16:49
  • 2
    I see. But the procedure described [in the thread](http://stats.stackexchange.com/questions/14099) you gave the link to, uses multiple test sets too: the only difference from k-fold is that the training data should be only preceding the test chunk. If you adopt this approach then you can obtain several error curves too. My main point is that one of the reasons you might be getting local minima in your error curve is due to noise, so it would make sense to try to reduce it by additional estimates. – amoeba Dec 15 '15 at 17:17
  • @amoeba thanks for your comment. So, if I'm following you, you suggest that instead of simply computing a single one-step ahead error, I can take the whole test chunk and compute an error metric. Then keep rolling forward (as the method suggests) and do the same for each new (and smaller) test set. Finally, average all the different errors to get the CV error. Your idea sounds right to me, but I just want to confirm if what I described is what you're talking about. Also, do you think the change in size in the test set affects the CV error as the rolling goes on? – jroberayalas Dec 15 '15 at 18:29
  • I have no experience with such a time-series-adapted CV, and I was not even aware of it until you provided the link to that other thread. So I am simply repeating what I read there. It looks to me that [both answers](http://stats.stackexchange.com/questions/14099) actually recommend having the fixed test set size that is being shifted forward along the time-series. So it's the training set that is changing size, not the test set. Take a look at those two answers, they illustrate it. – amoeba Dec 15 '15 at 19:51
  • [Cont.] Sorry, perhaps there is some misunderstanding. "I computed the error on the fixed test set of 12 observations, but using the approach described here" -- can you explain what *exactly* you did? To me it does not seem that Rob Hyndman recommends to use a fixed test set. – amoeba Dec 15 '15 at 19:56
  • @amoeba I updated my post to include a detailed version of the steps that I followed. Hope it is clearer now. Sorry for any misunderstanding. – jroberayalas Dec 15 '15 at 21:37

1 Answers1

2
  1. Is it common for a cross-validation plot to have several minima? Shouldn't it be smoother with a single optimum?

Yes, this is very common. Adding an additional term improves accuracy proportional to how much information it adds, and decreases accuracy because the model has become more complex and harder to fit. Plus, there is noise due to the fact that your training set is finite. So you shouldn't expect this to be super smooth. I often smooth CV plots before trying to interpret them or find minima.

  1. Why use all 47 terms when it seems that the inclusion of some of them increased the cross-validation error?

It's not clear from the plot if any of the terms actually increased the CV error in a significant way, since there is some level of statistical noise. But you're right, maybe we shouldn't use all 47 of the terms - maybe it would be better to skip some of the functions in the middle. Selecting the exact set of features that gives the best performance is NP-hard, but you could try a greedy approach. First try each function by itself, and pick the one that gives the lowest error. Then try adding in every remaining function one at a time, and pick the one that decreases the error the most. Repeat, and stop once none of the remaining functions decreases the error. This isn't guaranteed to work (for example, maybe some functions are only useful as a group and not individually) but could give you a better set than just taking the functions in order.

  1. Why not use the first or the second local minimum as the best number of terms?

These have worse CV error than 47, so they are unlikely to be better models. However they might be informative - the functions on the downward slopes of these minima might be the best features to use, while the ones on the upswing are just adding complexity and hurting you. You could try taking the set of all functions of the 70 that decreased CV error when they were added, which could be better than your 47.

Chris
  • 1,111
  • 5
  • 6
  • Thank you very much for your reply, Chris. I just have a quick question, can you clarify the approach you follow to smooth CV plots? – jroberayalas Dec 18 '15 at 15:29
  • 1
    I've used a [Savitzky-Golay](https://en.wikipedia.org/wiki/Savitzky%E2%80%93Golay_filter) in [MATLAB](http://www.mathworks.com/help/curvefit/smooth.html), but any reasonable filter would work. Often my goal is to make sure the minimum I select is robust to small changes in a hyperparameter (a little different than your situation here) but it would also help for eyeballing things. – Chris Dec 21 '15 at 01:43