0

I'm using SVM and (neural network) for a time series prediction data-set in MATLAB R2016a with 800 samples. Currently I'm using 10-fold cross validation and grid search to find best SVM parameters. I'm using 90 samples (after this 800 samples) as out-of-sample to check performance of final model using best SVM (and neural network) parameters and training my model on whole first 800 samples.

The test accuracy of final model (10-fold cross validation) is about 98% (sensitivity and specificity of about 98%) but when I check designed model on last 90 out-of-sample data (which trained using whole first 800 samples) I have a poor accuracy (about 55~59% total accuracy, sensitivity and specificity). This is daily forecasting of a financial market. Why I have this behavior? I checked normal k-fold cross validation and sliding window validation (discussed below in comments). I had mentioned behavior (poor out-of-sample accuracies) in two methods.

user2991243
  • 3,621
  • 4
  • 22
  • 48
  • 1
    accuracy is not a proper scoring rule, so it will select the wrong model with high probability. Use a proper scoring rule like log-likelihood or brier score. – Sycorax May 16 '16 at 13:49
  • @C11H17N2O2SNa. I got same results with `bier score` in SVM model. why? any suggestion? – user2991243 May 16 '16 at 14:58

1 Answers1

1

You need to use time-series cross-validation to tune your models, rather than random cross-validation.

e.g. (borrowing from Rob Hyndman's blog):

library(fpp) # To load the data set a10
plot(a10, ylab="$ million", xlab="Year", main="Antidiabetic drug sales")
plot(log(a10), ylab="", xlab="Year", main="Log Antidiabetic drug sales")

k <- 60 # minimum data length for fitting a model
n <- length(a10)
mae1 <- mae2 <- mae3 <- matrix(NA,n-k,12)
st <- tsp(a10)[1]+(k-2)/12

for(i in 1:(n-k))
{
  xshort <- window(a10, end=st + i/12)
  xnext <- window(a10, start=st + (i+1)/12, end=st + (i+12)/12)
  fit1 <- tslm(xshort ~ trend + season, lambda=0)
  fcast1 <- forecast(fit1, h=12)
  fit2 <- Arima(xshort, order=c(3,0,1), seasonal=list(order=c(0,1,1), period=12), 
      include.drift=TRUE, lambda=0, method="ML")
  fcast2 <- forecast(fit2, h=12)
  fit3 <- ets(xshort,model="MMM",damped=TRUE)
  fcast3 <- forecast(fit3, h=12)
  mae1[i,1:length(xnext)] <- abs(fcast1[['mean']]-xnext)
  mae2[i,1:length(xnext)] <- abs(fcast2[['mean']]-xnext)
  mae3[i,1:length(xnext)] <- abs(fcast3[['mean']]-xnext)
}

plot(1:12, colMeans(mae1,na.rm=TRUE), type="l", col=2, xlab="horizon", ylab="MAE",
     ylim=c(0.65,1.05))
lines(1:12, colMeans(mae2,na.rm=TRUE), type="l",col=3)
lines(1:12, colMeans(mae3,na.rm=TRUE), type="l",col=4)
legend("topleft",legend=c("LM","ARIMA","ETS"),col=2:4,lty=1)
Zach
  • 22,308
  • 18
  • 114
  • 158
  • Thank you for answer. I used both normal and your proposed cross validation techniques but I have same behavior. – user2991243 May 16 '16 at 13:56
  • I used this technique: fold 1 : training [1:n-50], test [n-50:n-40] ~~ fold 2 : training [1+10:n-40], test [n-40:n-30] and so on. This is just an example with imaginary numbers. Is this the same method? If not, please describe more about your technique. I'm not familiar with R. – user2991243 May 16 '16 at 19:53
  • I checked porposed method in : http://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection but I have same problem. – user2991243 May 16 '16 at 21:27