Should I use 5KFold or 10 KFold CV for data about 300000 sample?

Question

I use logistics regression for binary classification. and my data is about 300 000 samples. I have used both kfold = 5 and kfold =10 cross-validation

With Kfold= 5,  I have got :
Model1: AUC Test = 0.65  
Model2: AUC Test = 0.69 



With Kfold= 10,  I have got :
Model1: AUC Test = 0.61 
Model2: AUC Test = 0.65 

Since both of those models give me the same difference (0.04) is there any rule that should I respect it for choosing a better number of K. i.e. which is better to use when we have data of that number of samples.

and does a biger number of metric can have any importance (0.69 5kfold / 0.65 10kfold)

Answer 1

There is already an excellent thread on CV.SE on: "Choice of K in K-fold cross-validation", I would suggest read through it carefully because it offers very good context.

Particularly to the setting of this question I would point out the following:

1. during model selection what we ultimately care for is the order rather than the absolute value of AUC. In that respect both procedures suggest that model 2 is better.

2. a. The choice of sample size relates to the underlying sample variability. If we expect extremely small variability even a tiny sample might be enough to have a reasonable estimate. Similarly even a huge sample might be inadequote for a very volatile phenomenon (e.g. hourly returns of FX markets).

   b. It would be more relevant to perform repeated $k$-fold cross-validation to get an estimate of the variability of the metrics reported. This is genueily userful because we do have variation due to random sampling. The variation should decrease as our sample size gets larger but it is always good to quantify the variance of our metrics.

Anecdotally: Under "usual circumstances", a sample size of $n \sim 300K$ should be as perfectly adequate for a logistic regression to be an efficient estimator. In a similar case use case, I did $2$-fold cross-validation 5 times and my results generalised beautifully when deployed.