1

I have built 2 models:
1) precision: 0.80 - AUC ROC: 0.69
2) precision: 0.90 - AUC ROC: 0.94

I have posted both them to Kaggle as Titanic competition, the first model scored 0.7 and the second one scored 0.4. So I know the second model has been over-fitted. How can I find it out before sending results to Kaggle using plots or python codes? Does CAP curve do that? Can I find it out using ROC curve?

I have used train_test split for the Kaggle training dataset.

Ali Majed HA
  • 135
  • 1
  • 6

3 Answers3

3

Short answer:

ROC won't help you. A robust approach is to apply k-fold cross validation and count how many times training set accuracy is better than test set accuracy. If training set "beats" test set in the majority of folds, then your model is most likely overfitting. Instead of majority voting, you can alternatively compare the average accuracy in all training sets to the average accuracy in all test sets.

Long answer:

For a more detailed answer see here.

Digio
  • 2,427
  • 12
  • 18
0

Create validation dataset(from the train dataset) and test your model on it before submitting. You can compare your model prediction against the validation labels to get the performance of the model. NOTE: model shouldn't be trained on the validation data

Shudharsanan
  • 117
  • 2
  • 6
0

The best way to check if your model is overfitting or underfitting is the loss error. Usually they have a shape like this one 1.

This one

If the validation loss is increasing but your training loss is still decreasing is that you model is not generalizing and it is overfitting. This one of the best ways to check if so.

To avoid this there are multiple ways to do during training like dropout of batch normalization... Or as Digio said, K-Fold cross validation is another good way if you lack data.

Guillem
  • 41
  • 4