Using Rolling Forecast Origin Resampling in R for Neural Network Time Series

Question

I am new to time series prediction and forecasting with neural networks and am having trouble with cross validation.

I am fitting a multivariate time series. I have 236 monthly observations. I am using the caret package for this, seeing as Rob Hyndman has suggested to use rolling forecast cross validation for time series here: https://www.otexts.org/fpp/2/5.

Caret has the function to do so but, for starters, I am having some trouble understanding the ins and outs of the trainControl documentation.

What does the initialWindow mean in layman's terms? From the documentation on time slices it says the initial number of consecutive values in each training set sample.

I figured, using Rob Hyndman's approach, that I would use 235 of my total 236 observations, the last observation being the "test set", thus setting initialWindow to 235.

Here is the code I used:

control <- trainControl(method        = "timeslice", 
                        initialWindow = 235,  
                        fixedWindow   = TRUE, 
                        horizon       = 1)
mynn <- train(mytsframe4[,c(2:3)], mytsframe4[,1], 
              method    = "mlp", 
              size      = 2,
              metric    = c("RMSE"), 
              maximize  = FALSE, 
              trControl = control)
mynn

Multi-Layer Perceptron 

236 samples
  2 predictor

No pre-processing
Resampling: Rolling Forecasting Origin Resampling (1 held-out with a fixed window) 
Summary of sample sizes: 225, 225, 225, 225, 225, 225, ... 
Resampling results across tuning parameters:

  size  RMSE           Rsquared  RMSE SD      
  1     0.05837386693  NaN       0.04002651320
  3     0.05759843218  NaN       0.04774038998
  5     0.07597407274  NaN       0.03000920417

The data has been normalized, no missing values, from Feb 1995 to Sept 2014.

In summary, here are my questions I've already typed and a few more:

What does the initialWindow parameter mean in layman's terms?

What does the fixedWindow mean in layman's terms?

How is the output of the model interpreted? More specifically, what does the "size" mean?

Why could be causing NaNs in the Rsquared column?

How do I obtain the outputs/predictions so I can create unscaled forecasts?

Answer 1

I'll try to answer to your questions.

What does the initialWindow mean in layman's terms?

To better understand this I suggest to have a look at https://rpubs.com/crossxwill/time-series-cv. Put in simple terms, the initialWindow is the size (number of samples or row's dataset) of the training dataset used by the MLP (Multi-Layer Perceptron neural network) for its training stage used for each resample.

What does the fixedWindow mean in layman's terms?

This means that for each resample, the size of the training dataset does not change but remain the same; it's simple shifted onwards by an amount specified by horizon (in your case set to 1, i.e. a one step-ahead forecast)

How is the output of the model interpreted? More specifically, what does the "size" mean?

Size is the number of neurons in the hidden layer of the MLP. You specified size but it has been neglected in the way you set it in your code.

Why could be causing NaNs in the Rsquared column?

Try to use preProc = c("scale","center") in train.

How do I obtain the outputs/predictions so I can create unscaled forecasts?

Use:

mynn$Prediction <- predict(mynn, newdata = YourNewTestSet)

I also suggest to consider the following topic (Whether preprocessing is needed before prediction using FinalModel of RandomForest with caret package?)

I hope it was useful.