2

So I got my training set with 70% of my data called "train" / 30% "test"

I use ctree to get my decision tree model with something like this code below : 

model_ctree <- ctree(response ~ x1 + .. xn , data = train)

How can I apply this model to "test" and evaluate the model, use something like lift or gain chart or ROC; something that I would normally get from SAS miner?

I am new to R.

cardinal
  • 24,973
  • 8
  • 94
  • 128
JPC
  • 185
  • 1
  • 2
  • 8
  • Dear JPC, welcome to cross validated. I voted to move your question to stackoverflow, because it is not asking about statistics but about programming (you already know you want an ROC as opposed to asking what ROCs are good for). – cbeleites unhappy with SX Feb 06 '13 at 18:23
  • Also, please tell us what resources you used to look up what you are looking for, and get some introductions/tutorials on R. Many of them will tell you that R has help searching functions and packages that you can use to search e.g. for "ROC". Particularly, have a look at package "sos". Also, virtually any introduction that tells you how to fit a model a predictive model will also tell you how to use it to predict new cases. – cbeleites unhappy with SX Feb 06 '13 at 18:23
  • I can't tell yet whether this question belongs on SO or here. Are you wondering which method (eg gain chart vs ROC) would be optimal, or just how to get R to perform that method? If the former, I think it should stay here; if both, I think it could go to either, based on your preferences. (I'm guessing you'd prefer here, since you asked it here.) We only need to migrate if it's just the latter. – gung - Reinstate Monica Feb 06 '13 at 19:15
  • HI guys, thanks for the response first of all :) @gung : I do want some recommendation as far as what type of model evaluation I should use in this case. I will move this to SO. – JPC Feb 06 '13 at 20:18
  • You're welcome to move it to SO, if you like, @JPC, but if you want recommendations about model evaluation, I would say it could stay here. – gung - Reinstate Monica Feb 06 '13 at 21:28
  • 1
    Welcome to this site, JPC. Please, don't cross-post but rather ask a moderator (by flagging your post) to migrate your question. The [duplicate on SO](http://stackoverflow.com/q/14738037/420055) is unlikely to be well received as it stands, but see [this discussion](http://meta.stats.stackexchange.com/q/1335/930) on our Meta. I agree with @gung that this question could well be on-topic here if emphasis was put on the statistical aspect of model evaluation rather than programming. – chl Feb 06 '13 at 22:07
  • JPC, can you please review the reply you got on CV and decide where you want to keep this thread. [Cross-posting](http://meta.stackexchange.com/q/64068/150510) is frowned upon on Stack Exchange. – chl Feb 08 '13 at 08:19

1 Answers1

7

Try this for class predictions:

pred <- predict(model_ctree, newdata=test)
library(caret)
confusionMatrix(pred, test$response)

Try this for class probabilities:

probs <- treeresponse(model_ctree, newdata=test)
pred <- do.call(rbind, pred)
summary(pred)

Try this for a roc curve:

library(ROCR)
roc_pred <- prediction(pred[,1], test$response)
plot(performance(roc_pred, measure="tpr", x.measure="fpr"), colorize=TRUE)

Try this for a lift curve:

plot(performance(roc_pred, measure="lift", x.measure="rpp"), colorize=TRUE)

Sensitivity/specificity curve and precision/recall curve:

plot(performance(roc_pred, measure="sens", x.measure="spec"), colorize=TRUE)
plot(performance(roc_pred, measure="prec", x.measure="rec"), colorize=TRUE)

More info:

?ctree
?confusionMatrix
?performance

Also, you should check out the caret package if you're building predictive models in R. It implements a number of out-of-sample evaluation schemes, including bootstrap sampling, cross-validation, and multiple train/test splits. caret is really nice because it provides a unified interface to all the models, so you don't have to remember, e.g., that treeresponse is the function to get class probabilities from a ctree model. Here's an example of using 10-fold cross-validation to evaluation your model, which is much better than a single train/test split:

model <- train(response ~ x1 + .. xn , data = train, method='ctree', tuneLength=10,
               trControl=trainControl(
                 method='cv', number=10, classProbs=TRUE, summaryFunction=twoClassSummary))
model
plot(model)
Zach
  • 22,308
  • 18
  • 114
  • 158