1

I am developing a probability to default model on a data from landing firm. After running the GLM() model i have got the below message:

Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

Then i run xgboost() and was able to get decent accuracy. Now i want to determine the important features and their impact on customer default.

But not sure how to go ahead, as far as i know i could not get variable significance in xgboost() and the GLM() was run with above error.

So can't advise/conclude anything confidently.

Please Note : I am not looking for suggestions on how to avoid perfect separation problem(that is already available in many posts) but need help on how to advise business on change in which feature impact the default rate to what extent.

I know only GLM() model based on which i can give some advise but at the moment i am not so confident on glm() results, so what all other techniques can be picked up.

SKB
  • 155
  • 7
  • One method among many [tag:boruta] – Sycorax Dec 18 '19 at 14:16
  • As far as i know of boruta, it give the list of important variables but will i get the relative impact of change in one unit of independent variable on dependent variable(like we get in logistic regression)? – SKB Dec 19 '19 at 05:42
  • No, boruta won't tell you about 1-unit changes, but xgboost's default variable importance measurements won't tell you anything about 1-unit changes, either. If knowing about 1-unit changes is important, you should edit your question to explain that. – Sycorax Dec 19 '19 at 13:10
  • 1
    @SycoraxsaysReinstateMonica: Partially dependency plots for GBMs probably can answer that in reasonable manner. (Side-note: I have found PDPs annoyingly/disturbingly close to a GAM coefficient plots but you know... *ce la vie*!) – usεr11852 Dec 19 '19 at 15:39

1 Answers1

1

We can get variable importance from XGBoost (and gradient boosting procedures in general).

There are a few ways it can be computed (e.g. # of times a particular variable was used for splitting (commonly referred as Frequency), the total gains of splits which use a particular variable (commonly referred as Gain) and # of observations related to this features (commonly referred as Coverage)). R's xgboost package contains a method called xgb.importance that allows us to compute different feature importances in a model. That said, you might also want to explore the concept of Shapley values and how they can be used to find the most impactful features for a given model (and/or particular observation if needed). Assuming one is using R, the package iml is a great tool to first check.

I would also suggest exploring in a bit more detail the reason why glm returns this warning. Note that probabilities numerically equal to 0 or 1 are not categorical evidence that the GLM fitting procedure failed. I would suggest looking into this very informative CV.SE thread on: Unstable logistic regression when data not well separated. Aside that I would suggest considering the use of a regularised logistic regression (e.g. ridge regression through glmnet with alpha=0). Computationally, this allow the iterative model fitting procedure used by a GLM to exhibit "higher convexity", i.e. the ML procedure is able to converge to a minimum. Again, CV.SE has a great thread on this matter here: Regularization methods for logistic regression.

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • Thanks for the reply, with above suggestion partial problem will be solved & will get the important features but still i will not be able to determine the impact of these selected features on target variables. Like i will not be able to derive any conclusions on what will be the impact on target variable on unit raise in independent variable(like we have this in logistic regression). Also regarding problem in logistic regression(perfect separation), i will try glmnet again, as in first run it took hell lot of time. Can you tell me how can i figure out the variables causing perfect separation? – SKB Dec 19 '19 at 07:32
  • I am glad I could help; if you find an answer helpful you could consider upvoting it. For your side questions: 1. In the case of GBMs we can use partial dependency plots to show how changes in a particular explanatory variable affect the outcome; the `iml` package mentioned has a relevant example (and functionality), it can be seen in the [vignette](https://cran.r-project.org/web/packages/iml/vignettes/intro.html) too. 2. It will most likely be explanatory variables associated with either huge $\beta$ coefficients and/or standard errors. – usεr11852 Dec 19 '19 at 13:09