0

Estimating the mean or expected value of a continuous random variable's (r.v.) empirical distribution is known to be difficult, moreso than estimating the variance. Estimates of the mean and variance are therefore considered to be prone to estimation error.

Entropy (for discrete r.v.'s) and differential entropy (for continuous) are sometimes considered to be measures that capture the entire statistical distribution of a r.v. and can outshine extensions to higher moments. But how reliable is it to estimate the entropy of an empirical distribution compared to the mean and variance of a distribution? Is entropy less prone to estimation error?

develarist
  • 3,009
  • 8
  • 31

1 Answers1

1

The answer is no, for at least two reasons.

First, while some continuous distributions have what's called a differential entropy, not all do.

Second, empirical point estimates of entropy have inherent bias. See this page and its links.

The basis for your statement:

Estimating the mean or expected value of a continuous random variable's (r.v.) empirical distribution is known to be difficult, moreso than estimating the variance. Estimates of the mean and variance are therefore considered to be prone to estimation error.

is not completely clear; a reference justifying that would be useful. The statement at first glance seems to be contrary to the central limit theorem and the law of large numbers. But even were the statement true, estimates of entropy certainly would have no advantage.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • there does seem to be conflicting beliefs about using entropy, such as the widespread belief that every distribution has an entropy whereas not every distribution has statistical moments. Some argue this some don't – develarist Aug 03 '20 at 15:39
  • 1
    @develarist the "widespread belief that every distribution has an entropy" isn't true for continuous distributions, as [this page](https://stats.stackexchange.com/q/479629/28500) documents. – EdM Aug 03 '20 at 15:50
  • turning to the second point about bias, could it be said that, at least entropy is not a high variance estimator, and bias is not as bad as variance? in terms of the bias-variance trade-off – develarist Aug 03 '20 at 18:11
  • @develarist a [manuscript linked in my answer](https://arxiv.org/pdf/cond-mat/0403192v3.pdf) goes into bias-variance tradeoffs for entropy estimates in terms of correcting for the known bias in the plug-in estimator. Don't know what advantage entropy would have when analyzing finite samples taken from discrete distributions. And entropy for a discrete distribution only depends on class probabilities, so any re-ordering of probabilities among a set of classes would give the same entropy. E.g., a Poisson-distributed RV would have the same entropy if P(n=0) and P(n=1) were interchanged. – EdM Aug 03 '20 at 19:46
  • only interested in entropy for continuous r.v. – develarist Aug 03 '20 at 20:07