1

Using the following R code I obtain a decision tree using the agaricus dataset:

data(agaricus.train, package='xgboost')

bst <- xgboost(data = agaricus.train$data, label = agaricus.train$label, max_depth = 3,
               eta = 1, nthread = 2, nrounds = 2,objective = "binary:logistic")
# plot all the trees
xgb.plot.tree(model = bst)
# plot only the first tree and display the node ID:
xgb.plot.tree(model = bst, trees = 0, show_node_id = TRUE)

I want to understand more clearly the "value" output of the tree (the 3rd line in the oval shaped object). Here we can see that tree 0 leaf 7 gives a value 1.90174532. (That is the first terminal node in the image). I want to know if this value is the same as the log-odds score. So, all observations which follow the upper path of the decision tree will obtain a log-odds score of 1.90174532. Then in a new decision tree the observations will fall into a different split depending on each observations characteristics and will obtain a "new" value Then we sum up all these values across all trees to obtain a final log-odds score which can then be converted to a predicted probability using the logistic function.

Is my intuition correct? Does value = log-odds.

( https://rdrr.io/cran/xgboost/man/xgb.plot.tree.html )

enter image description here

user113156
  • 283
  • 2
  • 17
  • 1
    I think you will find the information you need in the linked thread. Please read it. If it isn't what you want / you still have a question afterwards, come back here & edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. – gung - Reinstate Monica Dec 01 '18 at 02:05
  • Yes, thank you for the link. I understand it, I just have one additional question. – user113156 Dec 05 '18 at 17:21
  • The answer states 'Each node in the tree has an associated "weight."' So using the tree example in my original question each decision node will have a "weight" which is not displayed. That is node 0-0 will have a weight, node 0-2 will have another weight, node 0-5 will have an additional weight. Then the terminal node is a summation over all the "weights" preceding it, in nodes 0-0, 0-2 and 0-5 which depending on the side of the split you obtain a value (or final weight) of 0.8085 or -1.985. Is this correct? Finally..... – user113156 Dec 05 '18 at 17:29
  • XGBoost constructs another tree (tree 1) where the features are in different positions in the tree and the split numbers are also different. Then the observations will fall into different terminal nodes and because the features are in different positions in the tree the summation of all the "weights" in the terminal node will be different. So an observation which fell into terminal node 0-12 in tree 0 may now fall into terminal node 1-8 in tree 1. With two terminal scores/weights. This will go on for say 100 trees in our model, so each observation will have 100 terminal weight-scores.......... – user113156 Dec 05 '18 at 17:35
  • These weights/scores are then summed up and we obtain a log-odds score for each observation which can be converted to a predicted probability. - Is this correct? – user113156 Dec 05 '18 at 17:36

1 Answers1

2

The "value" is the contribution of a leaf to the logit. The logit for a sample is the sum of the "value" of all of a sample's leafs. Because XGBoost is an ensemble, a sample will terminate in one leaf for each tree; gradient boosted ensembles sum over the predictions of all trees.

Then the logit can be used in the ordinary way, such as computing the predicted probability of class membership.

More information about gradient boosted trees generally and XGBoost specifically can be found in https://arxiv.org/abs/1603.02754 or .

Sycorax
  • 76,417
  • 20
  • 189
  • 313