Saving model with highest accuracy or with least loss?

Question

I'm confused. I used to save the model with the best accuracy in validation. But now I'm wondering if this was wrong all the time and I should have saved the model with the least loss. On the other side...these two have to correlate with each other, am I right? What is correct?

Answer 1

It might be one of them, both, or neither. You didn't give us any specific problem, so no precise answer can be given, so let me share some loose comments. First of all, as already noticed in the comments, accuracy is a rather poor metric. It might be useful in some cases as a cheap and dirty metric, but usually people do not use it as a decisive metric. In general, unless you are working on Kaggle competition problem, your aim is not to minimize a metric, and the metric is not given to you. Usually your aim is to solve some kind of problem (reduce churn, improve customer satisfaction, make machine more energy-efficient, etc.), often those problems are not directly measurable, and for sure they do not have much in common with "reducing squared error", or "improving AUC/ROC". Commonly, you would have multiple metrics related to the real-life problems, you may be using combination of them, give them different priorities, use them to set up thresholds that need to be met by your model, or the metrics may even evolve over time. There's nice talk by Josh Tobin among the materials from the free, online Full Stack Deep Learning course that discusses use of metrics in machine learning projects, that I can recommend you for more details.

TL;DR the metric you would use would be the metric that is most relevant to your problem.