(Non-)linear regression at leafs decision tree

Question

Is it common to have a different regression technique at the leaves of a regression tree (for instance linear regression)? I've been searching for it for the past hour but all I find are implementations that have a constant value at the trees' leafs. Is there a reason why this is/is not common?

Answer 1

There has been quite some research on this topic over the last decades, starting with the pioneering efforts of Ciampi, followed by Loh's GUIDE, and then also Gama's functional trees or the model-based recursive partitioning approach by us. A nice overview is given in @Momo's answer to this question: Advantage of GLMs in terminal nodes of a regression tree?

Corresponding software is less widely used than simple constant-fit trees as you observe. Part of the reason for this is presumably that it is more difficult to write - but also more difficult to use. It just requires more specifications than a simple CART model. But software is available (as previously pointed out here by @marqram or @Momo at: Regression tree algorithm with linear regression models in each leaf). Prominent software packages include:

• In the Weka suite there are M5P (M5') for continuous responses, LMT (logistic model trees) for binary responses, and FT (functional trees) for categorical responses. See http://www.cs.waikato.ac.nz/~ml/weka/ for more details. The former two functions are also easily interfaced through the R package RWeka.

• Loh's GUIDE implementation is available in binary form at no cost (but without source code) from http://www.stat.wisc.edu/~loh/guide.html. It allows to modify the details of the method by a wide range of control options.

• Our MOB (MOdel-Based recursive partitioning) algorithm is available in the R package partykit (successor to the party implementation). The mob() function gives you a general framework, allowing you to specify new models that can be easily fitted in the nodes/leaves of the tree. Convenience interfaces lmtree() and glmtree() that combine mob() with lm() and glm() are directly available and illustrated in vignette("mob", package = "partykit"). But other plugins can also be defined. For example, in https://stackoverflow.com/questions/37037445/using-mob-trees-partykit-package-with-nls-model mob() is combined with nls(). But there are also "mobsters" for various psychometric models (in psychotree) and for beta regression (in betareg).

Answer 2

MARS does this

I think it isn't more popular is that a lot of the robustness of ensembles of decision tree style models comes from the fact they allways predict constant values in range they've seen.

Outliers in the data generally just get lumped together with the highest/lowest normal values in the data on the last leaf and don't cause strange predictions or throw off coeffichents.

They also don't suffer from issues with multicolinearity as much as linear models.

You mighe be able to adress these issues in an implementation but its probally easier and more robust to just add more trees in an ensemble via boosting or bagging untll you get the smoothness you need.

Answer 3

I found a method that does just this (a decision tree, where the leafs contain a linear-regression instead of an average value). They are called model trees [1] and an example is the M5P[2] algorithm of weka. In M5P a linear regression is at each leaf.

Edit: I found another package/model that does something similar and seems to give very good results for my datasets: cubist. An implementation in R is given by the cubist package [3]. Cubist adds boosting ensembling to M5P and what it calls 'instance based corrections'.

[1]: Torgo, L. Functional models for regression tree leaves. In Proceedings of the 14th International Conference on Machine Learning, pp. 385–393. Morgan Kaufmann, 1997.

[2]: M5P http://weka.sourceforge.net/doc.dev/weka/classifiers/trees/M5P.html

[3]: Cubist model Cubist: Rule- And Instance-Based Regression Modeling https://cran.r-project.org/web/packages/Cubist/index.html