12

I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum entropy, what kind of distribution would we use with if we only knew the distribution's mean and the mean absolute deviation?

Or does it make more sense to use the median and the mean absolute deviation from the median?

I found a paper Maximum Entropy Principle with General Deviation Measures by Grechuk, Molyboha and Zabarankin which appears to have the information I am curious about, but it is taking me a while to decipher it.

Nick Stauner
  • 11,558
  • 5
  • 47
  • 105
Dietrich Epp
  • 223
  • 1
  • 6

1 Answers1

15

These wise gentlemen, Kotz, S., Kozubowski, T. J., & Podgorski, K. (2001). The Laplace Distribution and Generalizations: A Revisit with Applications to Communications, Economics, Engineering, and Finance (No. 183). Springer.

challenge us with an exercise:

enter image description here

The proof can follow the Information-Theoretic proof that the Normal is maximum entropy for given mean and variance. Specifically: Let $f(x)$ be the above Laplace density, and let $g(x)$ be any other density, but having the same mean and mean absolute deviation. This means that the following equality holds:

$$E_g(|X-c_1|)=\int g(x)|x-c_1|dx=c_2 =\int f(x)|x-c_1|dx =E_f(|X-c_1|)\qquad [1]$$ Now consider the Kullback-Leibler Divergence of the two densities:

$$0\le D_{KL}(g||f) = \int g(x)\ln\left(\frac {g(x)}{f(x)}\right)dx = \int g(x)\ln g(x)dx -\int g(x)\ln f(x)dx \qquad [2]$$

The first integral is the negative of the (differential) entropy of $g$, denote it $-h(g)$. The second integral is (writing explicitly the Laplacian pdf)

$$\int g(x)\ln[f(x)]dx = \int g(x)\ln\left[\frac{1}{2c_2}\exp\left\{-\frac 1{c_2} |x-c_1|\right\}\right]dx $$ $$=\ln\left[\frac{1}{2c_2}\right]\int g(x)dx- \frac 1{c_2}\int g(x)|x-c_1|dx $$ The first integral integrates to unity, and using also eq. $[1]$ we obtain

$$\int g(x)\ln[f(x)]dx = -\ln\left[2c_2\right] - \frac 1{c_2}\int f(x)|x-c_1|dx = -(\ln\left[2c_2\right] +1) $$ But this is the negative of the differential entropy of the Laplacian, denote it $-h(f)$.

Inserting these results into eq. $[2]$ we have $$0\le D(g||f) = -h(g) - (-h(f)) \Rightarrow h(g) \le h(f)$$ Since $g$ was arbitrary, this proves that the above Laplacian density is maximum entropy among all distributions with the above prescriptions.

Alecos Papadopoulos
  • 52,923
  • 5
  • 131
  • 241