Understanding which features were most important for logistic regression

Question

I've built a logistic regression classifier that is very accurate on my data. Now I want to understand better why it is working so well. Specifically, I'd like to rank which features are making the biggest contribution (which features are most important) and, ideally, quantify how much each feature is contributing to the accuracy of the overall model (or something in this vein). How do I do this?

My first thought was to rank them based on their coefficient, but I suspect this can't be right. If I have two features that are equally useful, but the spread of the first is ten times as large as the second, then I'd expect the first to receive a lower coefficient than the second. Is there a more reasonable way to evaluate feature importance?

Note that I'm not trying to understand how much a small change in the feature affects the probability of the outcome. Rather, I'm trying to understand how valuable each feature is, in terms of making the classifier accurate. Also, my goal is not so much to perform feature selection or construct a model with fewer features, but to try to provide some "explainability" for the learned model, so the classifier isn't just an opaque black-box.

Answer 1

The first thing to note is that you don't use logistic regression as a classifier. The fact that $Y$ is binary has absolutely nothing to do with using this maximum likelihood method to actually classify observations. Once you get past that, concentrate on the gold standard information measure which is a by-product of maximum likelihood: the likelihood ratio $\chi^2$ statistic. You can produce a chart showing the partial contribution of each predictor in terms of its partial $\chi^2$ statistic. These statistics have maximum information/power. You can use the bootstrap to show how hard it is to pick "winners" and "losers" by getting confidence intervals on the ranks of the predictive information provided by each predictor once the other predictors are accounted for. An example is in Section 5.4 of my course notes - click on Handouts.

If you have highly correlated features you can do a "chunk test" to combine their influence. A chart that does this is given in Figure 15.11 where size represents the combined contribution of 4 separate predictors.

Answer 2

The short answer is that is that there isn't a single, "right" way to answer this question.

For the best review of the issues see Ulrike Groemping's papers, e.g., Estimators of Relative Importance in Linear Regression Based on Variance Decomposition. The options she discusses range from simple heuristics to sophisticated, CPU intensive, multivariate solutions.

http://prof.beuth-hochschule.de/fileadmin/prof/groemp/downloads/amstat07mayp139.pdf

Groemping proposes her own approach in an R package called RELAIMPO that's also worth reading.

https://cran.r-project.org/web/packages/relaimpo/relaimpo.pdf

One quick and dirty heuristic that I've used is to sum up the chi-squares (F values, t-statistics) associated with each parameter then repercentage the individual values with that sum. The result would be a metric of rankable relative importance.

That said, I've never been a fan of "standardized beta coefficients" although they are frequently recommended by the profession and widely used. Here's the problem with them: the standardization is univariate and external to the model solution. In other words, this approach does not reflect the conditional nature of the model's results.

Answer 3

A fairly robust way of doing this would be to try fitting the model N times where N is the number of features. Each time use N-1 of the features and leave one feature out. Then you can use your favourite validation metric to measure how much the inclusion or exclusion of each feature affects the performance of the model. Depending on the number of features you have this may be computationally expensive.

Answer 4

You are correct in your observation that merely looking at the size of the estimated coefficient $|\hat{\beta_j}|$ is not very meaningful for the reason mentioned. But a simple adjustment is to multiply the coefficient estimate by the estimated standard deviation of the predictor $|\hat{\beta_j}| \hat{\sigma}_j$ and use this as a measure of importance. This is sometimes called a standardized beta coefficient and in logistic regression it represents the change in the estimated log odds of success caused by a one standard deviation change in $x_j$. One issue with this is that it breaks down when you're no longer dealing with numeric predictors.

Regarding your last point, of course it's possible that a variable might contribute a lot to the estimated log odds while not actually affecting the "true" log odds much, but I don't think this needs to be too much of a concern if we have any confidence in the procedure that produced the estimates.

Answer 5

You are right about why you should not use the coefficients as a measure of relevance, but you absolutelly can if you divide them by their standard error! If you have estimated the model with R, then it is already done for you! You can even remove the least important features from the model and see how it works.

A more heuristic approach to study how different changes in the variables alter the outcome is doing exactly that: try different inputs and study their estimated probabilities. However, as your model is quite simple, I would usggest against that