3

I have posted a related Question based on a paper, Ma, Jian, and Zengqi Sun. "Mutual information is copula entropy." Tsinghua Science & Technology 16.1 (2011): 51-54.

In the paper, they say that mutual information $I(x)$ is related to copula entropy $H_c(x)$ via $I(x) = -H_c(x)$.

However, the sign of entropy is supposed to be non-negative, as is the sign of mutual information. This leads me to conclude that mutual information and copula entropy are zero. If mutual information is zero, then all variables are unrelated, so something here is borked.

In the linked question, ArnoV commented that differential entropy can be either positive or negative (or zero), and the R package by the paper authors appears to be for continuous variables (it does not like having tied values). Fine. I even have heard that copula theory gets a bit weird for discrete marginal variables (copulae are not unique), so perhaps we should restrict ourselves to the continuous case. Okay...

Then $H_c(x)\in\mathbb{R}$; entropy can be any real value. This then means that mutual information can be any real value.

But mutual information is not negative! This leads right back to both mutual information and copula entropy having to be zero. Contradiction! $\Rightarrow\Leftarrow$

The result in the paper is elegant and makes a lot of sense to me. If the copula describes the relationship between marginal variables and the mutual information quantifies the degree to which the marginal variables are dependent, they should be related. Yet the equation they give seems to result in math being broken.

What's going on? What am I missing?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Dave
  • 28,473
  • 4
  • 52
  • 104
  • I have also read that continuity of the distribution function(s) will guarantee that the copula is unique. – DifferentialPleiometry Oct 03 '21 at 03:31
  • 1
    I don't know if this is related, but differential entropy is not the continuous analogue of Shannon's entropy. Shannon assumed this to be the case in his original work, but it is not the case. – DifferentialPleiometry Oct 03 '21 at 03:34
  • "[As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities.](https://en.wikipedia.org/wiki/Differential_entropy#Variants)" – DifferentialPleiometry Oct 03 '21 at 03:34
  • MI is not entropy: it can be [expressed as *differences* of entropies.](https://en.wikipedia.org/wiki/Mutual_information#Relation_to_conditional_and_joint_entropy). There is no paradox here. It isn't any different than wondering why it's possible to locate points with negative values on a number line, despite the fact that all distances between such points are non-negative. – whuber Jan 01 '22 at 22:02

0 Answers0