4

I have about 200 observations and 33 predictors. Due to sample size limitation, I used an elastic net logistic regression model. I have really high specificity ~0.9 but really low sensitivity < 0.01. Out of curiosity, I run a logistic regression model with the same parameters as the elastic net. And I have high specificity ~0.9 and a much higher sensitivity of ~0.17. Why might this be the case? I used the same data and same predictors. I cross-validated my results at k = 10 folds. I thought the elastic net would be better at predictions?

Adrian
  • 1,665
  • 3
  • 22
  • 42

1 Answers1

4

First, making an reproducible example would be helpful to answer the question. With this level of details, it is very hard to tell what happened on specific data.

Second, I would suggest reviewing bias variance trade-off concept in machine learning. Here is a good resource.

It is perfectly OK that logistic regression out perform regularized logistic regression.

Regularization will make a "simpler" model. If un-regularized logistic regression is better than regularized version, it be indicative that the data is complex, using logistic regression may not be sufficiently to capture the pattern of the data, and if we add on regularization, it may be worse.

Some of my related answers.

How to know if a learning curve from SVM model suffers from bias or variance?

Regularization methods for logistic regression

Community
  • 1
Haitao Du
  • 32,885
  • 17
  • 118
  • 213
  • 1
    thanks for the response. It is difficult for me to provide a reproducible example, but I will keep trying. Could you elaborate on how the bias-variance trade-off concept is relevant here? – Adrian Jan 19 '17 at 19:59
  • 2
    You want to know what exactly regularization (L1 or L2 or Elasticnet) do to your model. You want to know currently you are overfitting or underfitting, and if you add regularization, it will make problem better or worse. – Haitao Du Jan 19 '17 at 20:33
  • So regularization can be bad? Under what circumstance? – Adrian Jan 20 '17 at 02:03
  • 1
    @Adrian yes, when you are under fitting, regularization can make it worse. – Haitao Du Jan 20 '17 at 02:35