5

I'm training a simple logistic regression classifier on top of a rich feature set of 512 features for a binary classification problem. The training set is 200 observations and the validation set is 50 observations. These sizes cannot be changed. Early stopping is used based on the validation set to prevent overfitting (which is of course highly likely on this small training dataset). There's a separate test set for evaluation afterwards. The datasets are fairly unbalanced with ~15% of observations being one class.

I'm experiencing a strange phenomena, where the validation loss will continuously go down during the optimization process, but the validation accuracy will worsen at the same time (i.e. also go down). Any suggestions for why this might happen?

The loss function is crossentropy.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
pir
  • 4,626
  • 10
  • 38
  • 73

3 Answers3

3

Cross entropy, or almost all of the losses we use for classification, are surrogates for the accuracy function (loss is positive if you're wrong, negative if you're right). We typically choose such functions, like hinge loss for example, either because they are convex, differentiable, or some combination of both. Why not just optimize accuracy? Because the accuracy function is intractable, so solving for some parameters that optimize it is not really something we can do efficiently or intelligently.

To put it simply, you're optimizing cross-entropy in hopes it gives you parameters that help your model be accurate. But that isn't a necessary nor a sufficient condition for accuracy. Common practice is to ignore your validation loss and focus on your target metric: accuracy.

Mustafa Eisa
  • 1,302
  • 9
  • 19
2

I disagree with the answer by @Mustafa S Eisa. First, logistic regression is not classification. Logistic regression is risk estimation, and for use it makes a difference if the positive class have an estimated probability of .55 or .95. In the first case, if you want a hard classification/decision, whatever way you choose the risk of error is large. Accuracy is not taking that into account, and is known to not be a proper score function.

That is way what you have observed is to be expected, sometimes: Accuracy and crossentropy is not measuring the same kind of loss, and so crossentropy cannot be a surrogate for accuracy. There are many posts on this site about validating logistic regression that could be helpful.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
0

You probably need more than LR. Add a few hidden layers before last layer.

Check out my explanation and a small but impactful illustration by ahstat at https://stats.stackexchange.com/a/424697/112987.

Diansheng
  • 229
  • 2
  • 5