1

My question is similar to this link Creating a "certainty score" from the votes in random forests?

I am trying to build a random forest for a binary response (1 & 0). Let's say we have 10,000 different records and I am building 500 trees. Is there a way to score the records in terms of the certainty / confidence / likelihood of being categorized in category 1 (for example)? The link above suggests using the number of votes among all 500 trees, but this way can only give me up to 500 different scores, how can I differentiate further for these 10,000 records? (Like regression, the scores can be easily obtained).

One solution is to average the score of each tree in the forest. the tree is the probability of 1s in the final node. Anyone know how to produce that average in R? I couldnt find this in the randomForest package. I think if I write my own codes to do that it , the run time may not be as fast as a built-in function.

mdewey
  • 16,541
  • 22
  • 30
  • 57
zhifff
  • 11
  • 1
  • 3
  • 1
    Is your goal essentially to be able to sort your records by P(category = 1), & you don't want any ties? – gung - Reinstate Monica Oct 28 '13 at 14:37
  • @gung yes I want to be able to sort by P(category=1). but I dont want to sort by # of trees identifying category 1. Thanks !! – zhifff Oct 28 '13 at 17:50
  • That's the only way to create a confidence score for classification. What you must check is if this confidence correlates with classification error. One strange idea would be to check this, and if it is true, run another random forest regression on the % of prediction for class 1 for example with something like the quantregforest package which generates a sort of confidence for the prediction error. – JEquihua Oct 29 '13 at 07:29
  • One solution is to average the score of each tree in the forest. the tree is the probability of 1s in the final node. Anyone know how to produce that average in R? I couldnt find this in the randomForest package. I think if I write my own codes to do that it , the run time may not be as fast as a built-in function. – zhifff Oct 29 '13 at 20:23
  • This question was already answered here: http://stats.stackexchange.com/questions/56895/do-the-predictions-of-a-random-forest-model-have-a-prediction-interval – means-to-meaning Oct 29 '13 at 20:53

1 Answers1

0

If the only problem you're trying to solve is to have more than 500 distinct prediction values, then this is most easily solved by just fitting more trees. Adding additional trees to a forest doesn't overfit. The only practical concerns with this method is that fitting additional trees can consume additional time or memory.

For more information see: Do we have to tune the number of trees in a random forest?

As an aside, randomForest uses binary leafs -- each leaf can only predict 0 or 1. Some other implementations, most notably sklearn, use "proportional leafs", which means that the leaf emits a value in $[0,1]$, the proportion of samples belonging to class $k$ for all classes $k=1,2,\cdots,K$. Since each leaf in a tree will likely have a different proportion belonging to class $k$, this provides another route to yield a more diverse range of scores for your samples.

For more information, see: How to make the randomforest trees vote decimals but not binary

Sycorax
  • 76,417
  • 20
  • 189
  • 313