I have a binary classification problem with a large class imbalance ( 1/100 ).
I am getting fair results using ensemble modeling.
I understand that one technique that could improve results is upsampling the minority class and / or downsampling the majority class.
I have found that when I upsample the minority class (only in the training set of course) to a ratio 1/10 my cross validation results improve substantially and my performance of the model fit on the entire training set on unseen data improves relative to cross validation and unseen performance prior to up sampling the training data.
Here's the question:
When we upsample the training set we effectively alter its composition to bias the model with hopes getting better performance on unseen data. However, cross validation performance on the (altered ) training set is no longer reflective of expected performance on unseen data.
How do you assess model overfitting in this situation since comparison of cross val performance on training no longer reflects/ approximates fully trained model performance on unseen data?
