Random forest interpretation

Question

I don't understand if in random forest the features mean that a variable is positively correlated with the probability or simply it means that the feature has an influence in the prediction.

For instance, I have a target variable called "churn", with 1 meaning churn happened.

The variable age=25 has a 25% weight in feature.

Does it mean that age=25 influences churn=1 or does it simply influence churn and could for instance influence more churn=0?

Closely related: http://stats.stackexchange.com/questions/164048/can-random-forest-be-used-for-feature-selection-in-multiple-linear-regression/164068#164068 Random Forest doesn't sort out linear correlations in the same way as a regression; what makes it so powerful is that it is estimating different models for different subsets of the data, and those models are all axis-aligned rectangles of the feature space. So a feature can have a linear correlation of 0 (as per the link) and still be an important feature to a RF model. — Sycorax, Jul 05 '16 at 17:18

score 1 · Answer 1 · answered Jul 05 '16 at 19:14

First off:

Does it mean that age=25 influences churn=1 or does it simply influence churn and could for instance influence more churn=0?

If age has a high variable importance, then it is important for distinguishing cases with churn = 1 vs. cases with churn = 0. It is meaningless to discuss its influencing one of two classes more than the other.

As General Abrial notes, RFs are great at modeling nonlinearities. Correlations are inherently linear. (At least, Pearson's is.) Thus, you can have a variable that is not at all correlated with the outcome, but is nevertheless highly important for predicting it, through a nonlinear relationship.

Let's look at an example. Here is a predictor xx that relates to our outcome in a nonlinear manner:

set.seed(1)
xx <- runif(1e3,-1,1)
yy <- 1-xx^2+rnorm(1e3,0,.1)
cor.test(xx,yy)
plot(xx,yy)

cor.test() tells us that there is essentially zero correlation between xx and yy ($r=-0.04$). Nevertheless, it should be obvious that xx is important in predicting yy.

Let's add a few more completely irrelevant predictors, fit a RF and look at variable importance:

library(randomForest)
foo <- runif(1e3,-1,1)
bar <- runif(1e3,-1,1)
baz <- runif(1e3,-1,1)
rf <- randomForest(yy~xx+foo+bar+baz)
rf$importance

    IncNodePurity
xx      72.535584
foo      8.724830
bar      9.061573
baz      9.054412

still one doubt, I saw a post where with featureContribution function (in the R package rfFC) a person had these results. fc = c(-0.031544542, -0.064272583, -0.02307187, -0.000213402, -0.040743263, -0.042137713, -0.080828973) Instead I'm using Python and I never saw a feature with negative values. So the question is, is it possible to have features with negative values (assuming that they a negative influence on predicting the 1 category of the target variable) or it was just an error? — progster, Jul 07 '16 at 07:37
Good question. I'm not familiar with that particular package. Its documentation would be the first place to look. — Stephan Kolassa, Jul 07 '16 at 21:00

Random forest interpretation

1 Answers1