1

Through an L0 and L2 regularization method I have developed a logistic model that has an intercept of around -3. Taking the exp() of that alone I get a probability of around 5%. So that is when other parameters in the model are zero (they cannot be negative).

My question: Is that wrong for a logistic model ? Should they go from 0 to 1 no matter what, and hence my model is completely wrong, or...?

Denver Dang
  • 787
  • 4
  • 15
  • Please specify what "they" refers to in "Should _they_ go from 0 to 1 no matter what...?" Do you know for a fact that the probability of the positive outcome would be 0 when all the parameter values are 0? As a probability of 0 would mean an odds ratio of 0 and an infinitely negative log-odds, if you know that for a fact then you might need to consider a different approach than logistic regression. – EdM Jan 19 '20 at 21:46
  • 1
    Actually your $0.05$ should presumably be interpreted as the odds, with a corresponding probability slightly less; it is certainly possible to have an intercept greater than $0$ and so `exp()` of that greater than $1$ but this again will be the odds rather than probability - if the intercept had been $+3$ then the odds would be about $20.86$ and the probability about $0.9526$. There is nothing wrong with $5\%$ in a particular case if that reflects reality – Henry Jan 19 '20 at 21:48
  • @EdM No, just that it is quantities that are not negative. And not from 0 to 1 either. 0-100 is not uncommon for these parameters at least. – Denver Dang Jan 19 '20 at 21:59

1 Answers1

2

A few thoughts:

  1. The intercept is the log-odds of a positive class membership when all categorical predictors are at their reference levels and continuous predictors are at values of 0. An intercept of -3 represents odds of 0.05 and a probability of 4.7% at those conditions.

  2. Predictor variables can have any reasonable values. The important trick is making sure that their values are linearly related to the log-odds of class membership.

  3. As a probability of 0 means infinitely negative log-odds, a simple logistic regression model will not be able to handle actual probabilities of 0 adequately. It's possible to have perfect separation in a logistic model, with some combination of predictor values exactly distinguishing cases with probabilities of 0 and 1. In that case penalization of the model coefficients is one way to proceed.

  4. Whether you are penalizing to deal with perfect separation or for other reasons, note that penalization seeks deliberately to trade off increased bias for decreased variance. So if you are using a penalized model you should expect some bias.

EdM
  • 57,766
  • 7
  • 66
  • 187