5

I'm working with Shannon, Tsallis and Rényi entropies. I need to normalize these entropies for comparison purposes. In Shannon's entropy you need only to divide by the log of the number of bins.

$$H(X) = -\sum_{i}\left({P(x_i) \log_b P(x_i)}\right)/\log_b(N)$$

where $N$ is the number of bins and $b$ the log-base (in Shannon is equal 2).

Edit: Also for Rényi it is $\log(N)$

I'm missing Tsallis.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Marco
  • 201
  • 2
  • 12
  • Shannon entropy has nothing to do with base 2. You can express it with any base (though for 2 you get bits as units; but many people use $e$, as it is easier for some calculations). – Piotr Migdal Jan 27 '15 at 12:34
  • 1
    @PiotrMigdal: "nothing to do with base 2" is perhaps misleading: as you say, base 2 is not compulsory but people in various fields regard it as standard or even most appropriate. – Nick Cox Jun 13 '16 at 14:49
  • @NickCox Sure, base 2 is convenient and common. However, it seems that OP thinks that its a defining feature of Shannon entropy (vs other entropies). – Piotr Migdal Jun 14 '16 at 08:40

1 Answers1

2

Tsallis and Rényi entropy is the same thing, up to some rescaling. All of them are functions of $\sum_i p_i^\alpha$, with the special case of $\alpha\to1$ giving Shannon entropy.

Look at Tom Leinster's "Entropy, Diversity and Cardinality (Part 2)", especially at the table comparing these properties.

In short:

  • Rényi entropies are in $[0, \log(N)]$,
  • Tsallis entropies (called there $\alpha$-diversities) are in $[0, (1-N^{1-\alpha})/(1-\alpha)]$,
  • $\alpha$-cardinalities are in $[1, N]$.

Also, one more way to go is to use:

  • 1/cardinality, in $[\tfrac{1}{N}, 1]$,
  • just $\sum_i p_i^\alpha$, in $[\tfrac{1}{N^{\alpha-1}}, 1]$.

The later two have the advantage that no matter what is the $N$, they always end up in $[0, 1]$.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Piotr Migdal
  • 5,586
  • 2
  • 26
  • 70