0

I am trying to understand if my results are overfitting or not. I have the following results, using different features for model building:

Model 1

Total classified: 4696
Score: 1.0 # from cross validation
Score length 3

Confusion matrix:
[[2348    0]
 [   0 2348]]



Logistic Regression 1
              precision    recall  f1-score   support

         0.0       0.96      0.97      0.97       585
         1.0       0.76      0.67      0.71        76

    accuracy                           0.94       661
   macro avg       0.86      0.82      0.84       661
weighted avg       0.94      0.94      0.94       661

and

Model 2

Total classified: 4696
Score: 0.65 # from cross validation
Score length 3

Confusion matrix:
[[2154  194]
 [  66 2282]]


    Logistic Regression 2
                  precision    recall  f1-score   support
    
             0.0       0.96      0.97      0.96       585
             1.0       0.73      0.68      0.71        76
    
        accuracy                           0.93       661
       macro avg       0.85      0.83      0.84       661
    weighted avg       0.93      0.93      0.93       661

It seems clear to me, looking at the model's 1 confusion matrix

   [[2348    0]
     [   0 2348]]

and at its score (1), that I am having a problem of overfitting. However, I would like to ask you the following questions, all related to this topic:

  • in the second model, I am getting a score of 65 and a confusion matrix not perfect. Would it be ok to say that it is not overfitting based on the other metrics in the contingency table (recall, f1 score,...) as they are not so far from that value? (the problem is a classification one, with imbalance data)
  • what about the accuracy in the contingency table?
  • is there anything else that I need to consider?

Thank you for all the answers and comments for clarifying this (challenging) concept.

V_sqrt
  • 83
  • 6
  • Closely related https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models – Sycorax Feb 23 '21 at 03:49
  • Thank you Sycorax. I had not read that. However, it is not clear to me if the score I got in model 1 (or 2) is related to the accuracy in the classification report (or to another measure) for detecting overfitting. – V_sqrt Feb 23 '21 at 19:35
  • If you divide a confusion matrix by the number of observations, the sum of the cells on the diagonal is exactly the accuracy of the model. A confusion matrix just displays the same information as accuracy, parceled out into different types of error. But it has the same weaknesses as accuracy, because it doesn’t tell you anything about the degree of error, so it has many of the same shortcomings as accuracy, too. – Sycorax Feb 23 '21 at 19:38
  • thanks, Sycorax, it makes sense. I was wondering indeed how to spot overfitting in my case. In the first model, the perfect confusion matrix is giving me information on that. But in the second case, where I get a score of 0.65, I would say that it is not overfitting. Maybe I am wrong and I cannot say anything from the information that I have – V_sqrt Feb 23 '21 at 19:47
  • What's the definition of overfitting? How would you use the information you have here to apply that definition? – Sycorax Feb 23 '21 at 19:48
  • based on the definition of overfitting (that should happen when an analysis is too close or too exactly to a particular data under study), it is clear from the first model, and this is letting me say that it would not be the case of the second model, where I am getting only 0.65. Similarly for the classification reports, where all the metrics show results below 0.95. I am not totally familiar with these concepts, so I may be wrong – V_sqrt Feb 23 '21 at 19:51

0 Answers0