1

Entropy of probability distributions is the weighted average of the log probabilities of each observation of a random variable. Does this mean that every random variable that has a probability distribution function (p.d.f.) can be measured with entropy? Are there types of pdf's that cannot be measured with entropy?

How about random variables that have a cumulative distribution function (c.d.f.) only? Does entropy apply somehow to those with only a cdf?

develarist
  • 3,009
  • 8
  • 31
  • 2
    A random variable with a PDF has a *differential entropy.* A discrete variable has an *entropy.* The two aren't the same things (despite sharing many properties and part of their name). – whuber Jul 29 '20 at 16:00
  • so by default, and what most people refer to as entropy, is actually differential entropy then, since entropy is more often spoken of for random variables? – develarist Jul 29 '20 at 16:20
  • 1
    I think most people know about Shannon's entropy. Many think that it works on all r.v.s, and that its expansion to continuous r.v.s is trivial, like moments, but it is not. – Aksakal Jul 29 '20 at 16:29
  • Entropy applies to discrete random variables (and their distributions). Differential entropy applies to continuous random variables (and their distributions). – whuber Jul 29 '20 at 16:47
  • 1
    So, for a random variable with a mixed type distribution, such as a mixture of an (absoultely) continuous, such as a normal or exponential, and some atoms, it is not clear how to define entropy. – kjetil b halvorsen Jul 29 '20 at 18:11
  • you mean multivariate distributions, and therefore copulae, cannot be measured for entropy, even if they are simply still unimodal? – develarist Jul 29 '20 at 18:15
  • 2
    With that comment you seem to conflate the very different meanings of *multivariate,* *unimodal,* and *mixture distribution,* making it almost impossible to formulate a reply. As stated before: discrete random variables (univariate or multivariate, it makes no difference) have entropies and continuous random variables (univariate or multivariate) have differential entropies. – whuber Jul 29 '20 at 18:41

2 Answers2

2

If you mean differential entropy for continuous distributions then there are distributions whose entropy is undefined. So, the answer is NO.

For instance, look at the Warning on p.21 and the Footnote 4 in Polyanskiy and Wu, “Lecture notes on information theory”:

Shannon was probably the first who came up with a notion of entropy in information theory. So, he came up with a formula for discrete entropy, then by analogy wrote the one for continuous randoms which is called differential entropy. However, it turned out not to be the literal expansion from discrete to continuous. The latter is a different equation. Hence, I say that there are at least two different entropy equations for continuous variables.

The entropy as concept was introduced long before Shannon in statistical mechanics and thermodynamics.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • didn't know about differential entropy. is the standard entropy the same as differential entropy? what are all the different types of entropy – develarist Jul 29 '20 at 16:17
  • @develarist, the definition (shannon's) that you gave in the question cannot be applied to continuous distributions. the probability of any observation is zero. so you have to come up with something different for continuous, I believe there are a couple out there – Aksakal Jul 29 '20 at 16:21
  • one comment says differential entropy is for random variables while yours said it's for continuous variables. does this mean that random variables are continuous variables and not discrete varaibles? – develarist Jul 29 '20 at 16:23
  • 1
    random variables can be continuous or discrete. the main difference is that discrete ones have countable set of outcomes, i.e. you can enumerate all outcomes with an positive integer, e.g. Bernoulli or Poisson. continuous r.v.s cannot be enumerated, an example is Gaussian or Beta distributions. you can define some information measure to both types, including an entropy, but the same equation won't work for both discrete and continuous, at least not in every case – Aksakal Jul 29 '20 at 16:25
  • the document's section on differential entropy you gave in the answer is quite long. can you list some differential entropy measures? – develarist Jul 29 '20 at 16:32
  • i updated my answer – Aksakal Jul 29 '20 at 16:36
  • @Aksakal: Shannon certainly had precursors, see the quotes in my answer at https://stats.stackexchange.com/questions/87182/what-is-the-role-of-the-logarithm-in-shannons-entropy/463828#463828 – kjetil b halvorsen Jul 29 '20 at 18:22
  • @kjetilbhalvorsen good to know! – Aksakal Jul 29 '20 at 20:37
  • **Yager's entropy** minimizes the distance of a probability distribution from the uniform distribution. Does it fall into the same class of entropy measures as **Shannon's entropy**, applicable to discrete random variables, or is Yager a type of **differential entropy** measure applicable to continuous r.v.'s? – develarist Jul 30 '20 at 00:58
  • @Aksakal this answer says not all variables have *differential entropy*. can we also answer the question of whether all variables have *Shannon entropy*? – develarist Oct 13 '20 at 15:06
1

Every random variable has a probability distribution. You can say a pdf is unknown, but I don't think you can say that it doesn't exist.

As long as you have a random variable, you can measure its entropy.

If pdf is unknown, you can estimate pdf. You can also estimate entropy directly.

For example, an estimator for entropy is this:

"Summing the log of the nearest neighbors distances: a simple estimator for the entropy from samples" enter image description here

Update:

Any random variable can be described by its cumulative distribution function, which describes the probability that the random variable will be less than or equal to a certain value [1]. But, there are also some random variables that doesn't have a pdf nor a pmf, for example Cantor distribution. So, you can also say that a pdf doesn't exist.

moh
  • 438
  • 5
  • 13
  • 3
    "PDF" refers to a *density.* Many random variables have no density functions. Are you perhaps using "PDF" where you meant "CDF"? – whuber Jul 29 '20 at 15:59
  • 2
    I second whuber's comment. The density doesn't have to exist (Cantor distribution, for instance), but the CDF will exist. – Dave Jul 29 '20 at 16:01
  • does the blue figure contain an occuring process, where clusters of the original data are gradually being formed? the scatter plots look like they have different numbers of observations... and what is the purple figure showing? i thought entropy is a scalar value – develarist Jul 29 '20 at 16:16
  • ah... didn't know about this Cantor distribution. Thanks for clarifying this. – moh Jul 29 '20 at 17:19
  • @develarist. Yeah, I think the blue one is like sampling more and more from the true distribution, which is the purple one. – moh Jul 29 '20 at 17:21
  • it looks like a strange distribution that isn't common, and that should be clustered. we're talking about density functions, but how would **clustering** be of value here? would you say that distribution has 5 modes? – develarist Jul 29 '20 at 18:12