Inconsistency in cross-validation results

Question

I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this data according to the cognitive task being performed (everything is labelled).

The issue is that I do not have a large amount of data (approximately 60 trials per session, 30 for each cognitive task) and I have 2 sessions. I am trying to train a Linear Discriminant Analysis (LDA) classifier to classify this data. The classifier is later to be used in real-time to give some form of output every number of samples.

I made use of a 5-fold cross-validation to measure the generalization error of my classifier. The problem is that when I run this 5-fold cross validation a number of times, the results I get are not constant at all. Instead there's a significant variation in the overall accuracy (for example, first 5-fold cross validation may yield an average accuracy of 80%, 2nd yields an accuracy of 65%, 3rd yields an average of 72% etc...). Is this normal? If not, what could be the causes?

score 2 · Accepted Answer · edited Apr 13 '17 at 12:44

2

You may have discovered the fact that LDA models can suffer from instability: slight changes in the training set lead to very different models.

There's a whole body of literature on the variance of resampling methods for validation. This may give you a good start.

See also here for a related discussion. (I'm sure there is a literature list somewhere around here, but I cannot find it right now)

Anyways,

Braga-Neto has been working a lot on this
There's a very nice technical report by Ron Kohavi
We also did a study about Variance reduction in estimating classification error using sparse datasets
And another one on Assessing and improving the stability of chemometric models in small sample size situations

edited Apr 13 '17 at 12:44

Community

1

answered Mar 06 '13 at 22:09

cbeleites unhappy with SX

34,156
3
67
133

would using some form of regularization scheme for the LDA diminish this problem? – Lunat1c Mar 07 '13 at 09:55
@user2036690: possibly, yes. Still I recommend that you check first whether instability is really the problem, and after regularization check that things actually improve. *If* the problem is instability and your regularization is successful, you should also see an increase in hit rate. – cbeleites unhappy with SX Mar 07 '13 at 11:35

Inconsistency in cross-validation results

1 Answers1