2

This is my first foray into machine learning, as I usually rely on statistical methods.

In the past I've used SVM and its worked, but this is usually with data that is classification (e.g. iris and spam data). Currently, I am trying to using SVM to predict hourly wind speed data. Recorded wind speed can run anywhere from 0 to 45 in this data set, so essentially 45 categories.

Below is my code, which all works completely fine. The data set I'm using is an xts object which runs from 2016/01/01 to 2017/04/31 (20,423 observations). Aside from wind speed, the data set also contains 15 predictor variables which I include in the model.

svmtrain <- window(wdsp.ts.sub, 
                   end = as.POSIXct("2016-03-31 23:00:00"))
svmtest <- window(wdsp.ts.sub,
                  start = as.POSIXct("2016-04-01 00:00:00"))

# run kernel SVM

fitsvm <- ksvm(wdsp ~ .,
               data = svmtrain)

predsvm <- predict(fitsvm, svmtest[,2:16])

svmtab <- table(svmtest$wdsp, predsvm)

The last line of code is how I would normally compute my accuracy table, but it doesn't really make sense in this case.

How can I compute how accurate this model is, especially in comparison with something more intuitive/easy to understand like an ARIMA?

hoaxasaurusrex
  • 57
  • 6

1 Answers1

1

You have a time series, as you also indicate by contrasting your SVM to ARIMA. There are a number of common accuracy measures for time series forecasts, including the or the .

In your case, I would go with the (R)MSE, since wind speeds are presumably asymmetrically distributed, and minimizing the MAE leads to forecasting the conditional median, not the mean, so your forecasts will be biased.

Here is the section on "Evaluating forecast accuracy" from the excellent free online textbook Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • 1
    Hi Stephan, thanks for your reply. In terms of which measures for accuracy, that's fine, and you can extract those from R using accuracy() from Hyndman's forecast package. However, your response has made me realise what I need to do, which is to simply convert the svm forecasts to a ts object, and then use accuracy. Thanks! – hoaxasaurusrex Jul 24 '18 at 12:59