RFE, feature selection for Churn/Credit Risk modelling

Question

I am currently working on a redemption model for a financial company, using time series data and Logistic Regression. Currently we have a few features that are time dependant (I know, logistic is not the best for that). The current feature selection method we are going for is a wrapper RFECV that uses an algorithm called "GradientBoostingSurvivalAnalysis" which is a tree method.

The question I have is about which algorithm to use within the RFE, I know that there are many alternatives such as Gradient Boosting, Decision Tree, Random Forest, Logistic Regression, Cox PH, others. So far, I haven't seen a clear answer about when to use which, but in my mind using trees, for example, wouldn't be the best option since I am fitting the data using Logistic Regression, which is a "1 layer" algorithm, whereas trees uses many layers and the importance of features could vary depending on which layer you are, so it doesn't sound right to me (I could be really wrong). I think that a good solution could be going for a Survival Analysis approach and use, maybe, CoxPH regression or other survival method since these account for time-dependent variables and they are also a "1 layer" models.

What do you think?

Answer 1

It's probably a bad idea to use RFECV, as its Recursive Feature Elimination is generally very poor practice, often leading to results that don't generalize well to new data samples. Harrell's class notes provide a useful introduction to strategies for principled reduction of the dimension of the predictor space in the context of regression models, with much applicable to other modeling approaches. Critically, if your interest is in prediction you might be best off with including all predictors in your model provided that you take precautions like penalization (e.g., ridge regression) to avoid overfitting. For prediction, there's little to be gained by throwing away useful information.

That said, much of your question would remain if you were using an approach that doesn't involve automated feature selection.

I think that you might be making too strong of an analogy between depths of trees and levels in neural networks. The depths of trees in modeling approaches like gradient boosted trees just represent the number of interactions among predictors that are considered. That's no different from allowing interactions among predictors up to that depth in a "1 layer" logistic or survival model.

In your case, with what seems to be a single event (redemption) and time-varying covariates, a survival model would seem most appropriate. You are not, however, limited to standard survival regression approaches, as xgboost has some capacity for modeling parametric accelerated failure time models.