0

I know that log-loss penalises models that are confident with the wrong predicted classes. Can this be translated to percentage accuracy? If not, then how do I report the error or compare it to other percentage error metrics?

For example, on training a neural network with 128 output layers with sigmoid activation, I get a loss reduction from 0.30 to 0.04 over 20 epochs. How do I evaluate classifier accuracy based on this? It is a multi-label classification problem.

goelakash
  • 101
  • 1
  • 3
  • You don't, proper scoring rules don't relate to accuracy at all. There's a hint if you reach the numerical bottom or top of the scale though, but that's it. – Firebug May 29 '16 at 21:28
  • I read somewhere that it can be interpreted as e^(-total_log_loss). This gives an accuracy estimate (compared to a random guessing model with e^(-log(no_of_labels)). Is this correct or a useful interpretation? – goelakash May 29 '16 at 22:06
  • 1
    $e^{-\log{K}}=K^{-1}$ is the uniform random guess probability, but that only translates as an accuracy estimate if you assume a naive threshold for classification. On the other hand we have logloss $L = -\sum{\log{P_{i}}}$ and so $e^{-L}=e^{\sum{\log{P_{i}}}}=\prod{P_{i}}$, it takes a single $P_j=0$ to make it reach zero, even if you have all other $P_{i, i \neq j}=1$, and so that cannot be an accuracy estimate. – Firebug May 30 '16 at 14:39
  • Hmm, that makes sense. But then how is cross-entropy useful at all, if not as a proper metric? If it can't be reported, or compared, how does one define the viability of the model? (Lets say in terms of accuracy/ error rate of classification) – goelakash May 30 '16 at 16:14
  • It's useful because it penalizes too confident mistakes. It's actually one of the few proper metrics. It also can be reported, if you bound your probability outputs to say $[10^{-15}, 1-10^{-15}]$. Now this part is my opinion: accuracy/error rates are an unnecessary part of the data analysis, dichotomizing your scores/probabilities should be left to a decision-maker. In general, you want the best probabilities estimates possible, and logloss indicates exactly that. – Firebug Jun 13 '16 at 23:18

1 Answers1

-1

Convert Probability to LogLoss then back to Probablity

var probability = 0.5;
var logloss = Math.Log(probability);
Console.WriteLine(logloss); //-0.693147180559945
var originaProbability = Math.Exp(logloss);
Console.WriteLine(originaProbability); //0.5!

I think this is all it is

charlierlee
  • 1
  • 2
  • 2
    Could you explain how this might result in the "percentage accuracy" requested in the question? – whuber May 08 '19 at 22:05