5

I have a Bernoulli response variable and I am going to fit a logistic regression. One of my independent variables is a continuous random variable and I would like to categorize it before fitting the logistic regression. While this will lose some information, it makes my predictions a lot easier and at the same time I can see the effect of this continuous random variable easily. I am trying to categorize it such that each category would be distinct in terms of their performance on estimated probabilities. Ideally I would like to see the logistic regression coefficients of this categorized variable to be statistically significant. By experience, I know that the number of categorizes should be less than 8 as well. Most of the time it is around 4 or 5 categories. But the exact number of categories is actually unknown. Finding good break points is challenging here. I have tried Recursive Partitioning and Regression Tree before. But to use this approach, I would first need to categorize the independent variables myself and then it provides me with the breakpoints.

I was wondering if there is any other alternative approach to categorize this continuous independent variable.

  • Please note that this question is not asking on whether to categorize or not as I am aware of disadvantages and advantages of that. I hope those who want to answer or comment consider this before trying to convince me to not categorize it. Thank you.
Stat
  • 7,078
  • 1
  • 24
  • 49
  • 2
    [This page](http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous) may be helpful. You may also get better answers if you explain what you hope to achieve by discretizing your continuous predictor, and with no prior idea of how to discretize it, too. – Kodiologist May 23 '16 at 13:45
  • 4
    Categorizing continous variables does not make interpretation easier. Imagine you categorized human age in 10-year groups, you make predictions using your model and get different results for two persons who are the same except their age: one is 39 years old and another is 40 years old -- does this mean that there is a qualitative difference between those two age groups..? Does knowing that there is a "jump" in results for people who are 40+ make interpretation anyhow easier? – Tim May 23 '16 at 14:21
  • When you categorzie a variable, results can be interpreted and understood a lot easier for non-statisticians. In fact this is one of the reason people categorize variables. – Stat May 23 '16 at 14:28
  • 5
    If non-statisticican interpretation is your goal, then your partitioning should probably be optimized for ease-of-interpretation, not performance. Introducing a complicated algorithm for choosing cut-points will be nearly as difficult to explain to your audience as a complicated regression. – Andreus May 23 '16 at 14:41
  • 5
    This is a **bad** reason for doing so. You can easily manipulate the results and "lie with statistics" by changing the categories. – Tim May 23 '16 at 14:52
  • 1
    The usual arguments against discretization I gave here: [What is the benefit of breaking up a continuous predictor variable?](http://stats.stackexchange.com/q/68834/17230). But there's something else: you say both "I am trying to categorize it such that each category would be distinct in terms of their performance on estimated probabilities" & "Ideally I would like to see the logistic regression coefficients of this categorized variable to be statistically significant"; you need to bear in mind that the usual significance tests will be invalidated by using the response to guide discretization. – Scortchi - Reinstate Monica May 29 '16 at 15:49
  • (It's not entirely clear what you mean by "significance" anyway without specifying the coding.) By the way, ["Optimal binning"](http://stats.stackexchange.com/search?q=%22optimal+binning%22) is a common term for what you're asking about. – Scortchi - Reinstate Monica May 29 '16 at 16:26

4 Answers4

10

Instead of throwing away data by categorizing, you could consider fitting your continuous predictor as a spline function with a specified number of knots or with the number of knots chosen by cross-validation. That will use up no more degrees of freedom than categorization. If you are willing to envision up to 8 categories, it's not clear that categorization is really simpler than a well-modeled continuous variable, and predictions of new cases with the continuous fit should be better, too. Using spline functions in formulas with the rms package in R, as I recall, does this naturally; check the documentation.

Added in response to edited question and comments:

Non-statisticians might be better served by a set of illustrative examples drawn from a model based on the continuous predictor. You could choose examples so that they seem like categories ("very high","high", "medium", "low", "very low") even if the model doesn't itself depend on the categorization.

One situation where categorization in the model itself might be useful is if there really are distinct underlying classes of cases that your continuous estimator is obfuscating. With some effort such an example and some rationale can be found for a 2-class situation with high errors in measuring their 2 distinct values along a continuous scale, but it's hard to see how that would generalize to more than 2 classes.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • 1
    (+1) Have you got any more on the last point? (I remember reading a paper on this but have forgotten it.) – Scortchi - Reinstate Monica May 29 '16 at 16:34
  • 1
    @Scortchi I've done a simple 2-class simulation in which each class has mean (x,y) values of (0,0) and (1,1) with fairly large SDs. Playing with SDs and sample size can give a situation where linear regression with continuous x is "not significant" but using a cutoff in x gives a "significant" difference in y between the two groups defined by the cutoff. Takes some playing, but it can be done. Don't know any papers on the topic, but that's mostly a sign of my limited reading. Details must wait until I'm back from vacation and have access to my notes. – EdM May 29 '16 at 18:26
  • Thanks! I ought to ask that as a question. And to find the paper, which if I remember right agrees with what you describe. – Scortchi - Reinstate Monica May 29 '16 at 18:40
1

Since it seems like "ease of interpretation" is important to you, I think you would be interested to learn about nomograms, which are essentially a model represented in a diagrammatical way. Instead of relying on some ad hoc categorization procedure, you can fit ornate trends using statistically principled methods such as regression splines, and then represent the equation in the form of a nomogram. Predictions are made by drawing a line through the values of the predictor variables.

More information about both regression splines and nomograms can be found in Regression Modeling Strategies by Frank Harrell.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
0

I always think you can do most task by two approaches: knowledge driven and data driven include binning your continuous features.

  1. By knowledge driven, you can think about what binning will make sense from what the actual feature represents. For example, if you are binning a household income, you definitely can find some references on basic statistics of the US household income and use those statistical metrics to bin it (e.g., what is the typical value for middle class, rich etc.).

  2. By data driven, you are essential want to use this binning to improve your model performance. You can think about you are essentially doing feature engineering or basis expansion. Suppose you want to sacrifice your interpretability, you can even use Neural Network to "train the basis expansion", where you expand one continuous features to many "engineered features", those engineered features can be continuous or discrete. I am thinking you are using RPART to bin, is similar to this approach.

Best research always come with combining both knowledge driven and data driven, where you use knowledge to specify a "rough shape of the model" and use data to fit it to get more details. In your case of binning continuous variables, you may also do this.

I am not sure if my answer is too high level, but feel free to ask me to explain any part in details.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Haitao Du
  • 32,885
  • 17
  • 118
  • 213
-3

Thanks to those who tried to answer it. However, I don't think either of these answers are that much helpful to me. In fact there is a phd thesis written on this available here. There are also some R packages e.g. CatPredi that can be used as well.

Stat
  • 7,078
  • 1
  • 24
  • 49