12

The randomForest implementation does not allow sampling beyond the number of observations, even when sampling with replacement. Why is this?

Works fine:

rf <- randomForest(Species ~ ., iris, sampsize=c(1, 1, 1), replace=TRUE)
rf <- randomForest(Species ~ ., iris, sampsize=3, replace=TRUE)

What I want to do:

rf <- randomForest(Species ~ ., iris, sampsize=c(51, 1, 1), replace=TRUE)
Error in randomForest.default(m, y, ...) : 
  sampsize can not be larger than class frequency

Similar error without stratified sample:

rf <- randomForest(Species ~ ., iris, sampsize=151, replace=TRUE)
Error in randomForest.default(m, y, ...) : sampsize too large

Since I was expecting the method to take bootstrap samples when given replace=TRUE in both cases, I was not expecting this limit.

My objective is to use this with the stratified sampling option, in order to draw a sufficiently large sample from a relatively rare class.

smci
  • 1,456
  • 1
  • 13
  • 20
cohoz
  • 618
  • 5
  • 16
  • I'm not sure what the real reason is, but a bootstrap sample is typically the same size as your original sample, so this behavior seems perfectly in line with what I'd expect from something claiming to take bootstrap samples. – joran Dec 11 '12 at 15:20
  • 2
    Well, that was my word choice not the documentation's, but your point is well-taken. Still, this is inconvenient when trying to re-balance the distribution and I don't know a reason why this is a helpful restriction to impose. – cohoz Dec 12 '12 at 03:40

2 Answers2

5

This does not answer why, but to get around this, one can duplicate the data for the rare class in the training data, and take a stratified sample of the result.

Two drawbacks to this approach, compared with a "natural" oversampling:

  • the out of bag estimates are no longer meaningful
  • more resources are required to store the object and take random samples

but it will allow one to build the forest with the desired class ratios.

cohoz
  • 618
  • 5
  • 16
4

I have the exact same question and found this in the changelog for randomForest:

Changes in 4.1-0:

  • In randomForest(), if sampsize is given, the sampling is now done without replacement, in addition to stratified by class. Therefore sampsize can not be larger than the class frequencies.

Setting replace=TRUE manually also does not seem to override this.

hgcrpd
  • 1,307
  • 2
  • 11
  • 13
  • 2
    It may be the case that the replace parameter is simply being ignored, but later in that changelog: Changes in 4.5-12: * Added the 'strata' argument to randomForest, which, in conjunction with 'sampsize', allow sampling (with or without replacement) according to a strata variable (which can be something other than the class variable). Currently only works in classification. – cohoz Jan 24 '13 at 02:12
  • For example, the same error is generate via `rf – cohoz Jan 24 '13 at 02:18
  • Moreover, some test cases with the current version (4.6-7) indicate that samples are taken with replacement, so this isn't the explanation. – cohoz Jan 27 '13 at 17:09
  • 1
    I ended up just oversampling prior to running the randomForest. There's probably something wrong with this methodology, but it seem s to work when I test the results. – hgcrpd Jan 29 '13 at 02:15
  • 1
    No, that's the workaround that I use and it does exactly what one wants. Especially for the use case of one relatively rare class, duplicating the data for that class on the training set and before taking a stratified sample works well and the "cost" in terms of additional memory/CPU is not too high. I guess that's worth writing up an a "answer" even though it really isn't one... – cohoz Jan 29 '13 at 03:08