4



I am using the function svm from the package e1071 in R to generate a support vector machine model. I have a very large data set, and for the moment, while in an exploratory mode, want to simply read in small slices of the data that can be modeled on my single machine.

After obtaining the model results, I would like to read in more data and generate another SVM model, etc. until all of the data is modeled, which will generate about 50 different models. I would then like to merge all of these models / results together in some fashion in order to get a grand model which would approximate what I would achieve if I could fit all the data in at once.

I know there are multiple ways to do this, theoretically. But, sticking with SVM modeling, what are my options in R? (I want to stick with the e1071 package because it has some things about it that are not in the others that I saw.)

If it matters, my data is not genomic, and it is highly weighted (meaning that I am modeling T/F, and usually find an F, but the feature set data is not sparse).

Below is a "pseudocode" snapshot of what I'm proposing / hoping, in case that helps clear things up: 

require(e1071)
modelSet = NULL
for (i in c(1:50) ) { ## imagine there are 50 files
  dF <- read.csv(paste("file",i,"csv", sep=".") )
  modelSet[[i]] <- svm(myOutput ~ ., data= dF, probability= TRUE)
}
## Now I would like to find a way to merge all 50 of the "modelSet" models together
## to make 1 composite model derived from all 50 data files.



Thanks!
Mike

Mike Williamson
  • 432
  • 5
  • 16
  • 1
    what about using stochastic gradient descent to estimate your svm instead? how large is your feature set – seanv507 Jul 31 '13 at 23:38
  • EnsembleSVM may be of interest to you. It is made for exactly such situations. Unfortunately there is no R interface yet. http://www.esat.kuleuven.be/sista/ensemblesvm/ – Marc Claesen Aug 01 '13 at 13:45
  • Hi Sean, Marc. Thanks for the advice! Regarding feature set, it is about 85 features, but realistically about 20 that I care about. Each feature could have 1000+ levels, though. Marc, I will look through EnsembleSVM. Might be nice later down the road, but for now, I wanna stick with R to speed up development. Thanks! – Mike Williamson Aug 01 '13 at 21:56

3 Answers3

2

First, make sure you shuffle your data before you separate it in to the 50 partitions. If the original data is at all ordered, this can make a huge difference in the performance of your final model. Also, make sure your 50 partitions are stratified, meaning they have roughly the same number of exemplars per class as the overall data set. Then go ahead and train your 50 models (how you handle the hyperparameter selection is a bigger problem). When you want to make a prediction, do one of the following, all variations on boosting:

1) Hard-decision boosting: Evaluate all 50 models and treat the output of each model as a vote. Whichever class gets the most votes wins. The problem with this approach is that each model gets an equal vote, regardless of the confidence of its prediction.

2) Soft-decision boosting: Evaluate all 50 models with the parameter decision.values = TRUE. This tells e1071 to return the decision value in addition to the class label. Average the decision values returned by each of the 50 models and make a prediction based on the average. The problem with this approach is that the decision values will not necessarily be scaled equally across all models. In other words, a decision value of 1.618 will not necessarily mean the same thing across all 50 models.

3) Soft-decision boosting with logistic meta-model: Train and evaluate the models with the parameter probability = TRUE. This will fit a logistic regression on top of the SVM model that scales the output to be in the range [0, 1]. This is sort of a kludge but it may help with the scaling problem of option 2.

A final word of caution - you say there are only 20 features you care about but each feature can have 1000+ levels. By your use of the word "levels", I assume these features are discrete. Are they ordinal or categorical? In other words, is there an order to the levels? Is level 1 closer to level 2 than level 3? If so, you're probably good to go. If not, you might want to consider re-coding the categorical using one-hot encoding.

Bill Woessner
  • 1,168
  • 11
  • 15
0

I would offer you to combine predictions, not models. E.g., after you have 50 models, you can apply all of them on the testing data. Then you will get 50 prediction sets. And all this prediction sets you could use as an input to train another classifier/model (could be SVM as well).

This is described in more details in the "Practical Machine Learning" course on Coursera from Johns Hopkins University: https://www.coursera.org/course/predmachlearn.

There is a "combining predictors" video, and slideset for it is available, for example, here: https://rpubs.com/pr9115/JLeek_025_CombiningPredictors

Andrey Sapegin
  • 108
  • 1
  • 10
0

My solution is, If it is acceptable, better to compose all the observations into a single data set and go for stratified sampling, which might not be possible when data is separately in different files. Then build SVM on it.

Vamshidhar H.K.
  • 1
  • 2