2

Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer.

From wikipedia, we can see that joint entropy of two random variables are defined in terms of the probability of the possible outcomes: https://en.wikipedia.org/wiki/Joint_entropy

However, for the slightly more subtle definition of cross entropy, which is not symmetric, uses the concept of ditribution, but the notation is the same: https://en.wikipedia.org/wiki/Cross_entropy

This "misleading" notation for cross entropy is also shown on the wikipedia page for Kullback-liebler definition: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

To me, it looks like $H(X,Y)$ is fine for defining joint entropy, but what should I use then for cross entropy ?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Tobbey
  • 123
  • 5

1 Answers1

1

Note that this notation for cross-entropy is non-standard. The normal notation is H(p,q). This notation is horrible for two reasons. Firstly, the exact same notation is also used for joint entropy. Secondly, it makes it seem like cross-entropy is symmetric. This is ridiculous, and I’ll be writing Hq(p) instead.

Reference from link: http://colah.github.io/posts/2015-09-Visual-Information/#fn4

Also, possible duplicate of : https://math.stackexchange.com/questions/2505015/relation-between-cross-entropy-and-joint-entropy

Ayush Katyal
  • 84
  • 3
  • 1
    Relevant blog post, and the answer from the possible duplicate (although the initial question was different) makes sens. Writing $H_p(q)$ makes much more sense as it emphasize that the reference for optimal coding is p, and the target of the analysis in this framework is distribution q – Tobbey Oct 22 '18 at 10:57
  • Here's a new one on notation for cross-entropy: https://chris-said.io/2020/12/26/two-things-that-confused-me-about-cross-entropy/ – Louis Maddox Dec 26 '20 at 18:23