Does preclustering help to build a better predictive model?

Question

For the task of churn modelling I was considering:

1. Compute k clusters for the data
2. Build k models for each cluster individually.

The rationale for that is, that there is nothing to prove, that the population of subscribers is homogenous, so its reasonable to assume that data-generating process may be different for different "groups".

My question is, is it an appropriate method? Does it violate anything, or is it considered bad for some reason? If so, why?

If not, would you share some best practices on that issue? And another question: is it generally better or worse to do pre-clustering than model tree (As defined in Witten, Frank - classification/regression tree with models at the leafs. Intuitively it seems that decision-tree stage is just another form of clustering, but I don't know whether it has any advantages over "normal" clustering.).

Answer 1

There is a method called clusterwise regression that solves similar problem (first clusters data and then builts predictive models). See for example this.

Answer 2

Two points that are too long to be a comment:

• pure clusters (i.e. containing cases of one class only) are no problem per se: so called one-class classifiers model each class independent of all others. They can perfectly deal with this.

• However, if the data clusters in a way that the classes are quite separated, i.e. the clusters are rather pure, this means that a very strong structure exists, a structure that cluster analysis is able to find without guidance by the class labels. This means that certain types of classifiers such as nearest neighbour methods based on the same distance measure used by the cluster analysis are appropriate for the data.

• The other possibility, situations where the clusters are not pure, but a combination of cluster and classification methods can do well is appropriate for trees. The tree will do the part of the clustering (and pure nodes are not considered a problem.) Here's an artificial example, a 2 cluster version of the XOR-problem:
  

• another way to include the cluster information without running the risk of having pure clusters would be to use the clustering as a feature generation step: add the outcome of the cluster analysis as new variates to the data matrix.

• You ask whether it is bad for some reason: one pitfall is that this appoach leads to models with many degrees of freedom. You'll have to be particularly careful not to overfit.

• Have a look at model-based-trees, e.g. mbq's answer here I think they implement a concept that is very close to whar you look for. They can be implemented as forest as well: e.g. R package mobForest.

Answer 3

I'm dealing with similar problem these days. I have hundreds of feature to build classifier. After trying different models (ex: random forests, gradient boost, etc...), I still got low precision/recall. So I'm trying to do some clustering then build classifiers in different groups. My concern is, just like Anony-Mousse says, how can I gain more information from the classifier if I use all the information in clustering? So here's what I gonna do next:

1. Use some features (less, according to prior knowledge) to do clustering.
2. Use other features (more) to train classifiers.

I think it may also helps to reduce complexity, wish it helps.

Answer 4

Building $k$ clusters and then $k$ corresponding models is absolutely feasible. The pathologic case noted in the comments wherein the clusters perfectly separate the outcome variables would pose difficulties for classifiers is a theoretical problem, but one which I think is unlikely (especially in a high dimensional case). Furthermore, if you could build such clusters, you could then just use those clusters for prediction!

In addition, if the process begins with $N$ samples, the classifiers can only use $N/k$ samples. Thus, a more powerful approach would be to use the clusters in building a single classifier that incorporates the heterogeneity in the clusters using a mixture of regressions. In model-based clustering, one assumes the data are generated from a mixture distribution $Y_i \sim N(\mu_i, \sigma_i^2)$ where $i=1$ with probability $\pi$ and $i=2$ with probability $1-\pi$ and $\mu_1 \neq \ \mu_2$ and $\sigma_1^2 \neq \sigma_2^2$. A mixture regression is an extension that allows one to model the data as being dependent on co-variates; $\mu_i$ is replaced with $\beta_i X_i$, where the $\beta_i$ have to be estimated. While this example is for a univariate, Gaussian case, the framework can accommodate many data (multinomial-logit would be appropriate for categorical variables). The flexmix package for R provides a more detailed description and of course a relatively easy and extensible way to implement this approach.

Alternatively, in a discriminative setting, one could try incorporating cluster assignments (hard or soft) as a feature for training the classification algorithm of choice (e.g. NB, ANN, SVM, RF, etc.)

Answer 5

Well, if your clusters are really good, your classifiers will be crap. Because they have not enough diversion in their training data.

Say your clusters are perfect i.e. pure. You can't even properly train a classifier there anymore. Classifiers need positive and negative examples!

Random Forest are very successful in doing the exact opposite. They take a random sample of the data, train a classifier on that, and then use all of the trained classifiers.

What might work is to use clustering, and then train a classifier on every pair of clusters, at least if they disagree enough (if a class is split into two clusters, you still cannot train a classifier there!)