15

my question particularly applies to network reconstruction

  • This answer explains it from a fundamental point of view: https://stats.stackexchange.com/questions/81659/mutual-information-versus-correlation – Hamfry Nov 26 '20 at 18:08

2 Answers2

25

Correlation measures the linear relationship (Pearson's correlation) or monotonic relationship (Spearman's correlation) between two variables, X and Y.

Mutual information is more general and measures the reduction of uncertainty in Y after observing X. It is the KL distance between the joint density and the product of the individual densities. So MI can measure non-monotonic relationships and other more complicated relationships.

Rob Hyndman
  • 51,928
  • 23
  • 126
  • 178
  • 3
    Correlation is not necessarily linear - Spearman's rho relies on the monotonic function, and yet, we refer to it as a "correlation coefficient", not "mutual information coefficient". And for a good reason: it provides an information about association between two variables. Mutual information, redundant information, mutual variance, correlation - these terms are so similar, and this question refers to _network reconstruction_, so I guess that we ended up in the wrong area with right terminology. This is quite specific question... – aL3xa Jul 31 '10 at 02:26
  • 1
    Good point. I've edited my answer to include monotonic relationships. I don't know anything about network reconstruction. – Rob Hyndman Jul 31 '10 at 03:04
  • @RobHyndman Dear sir, can you please answer [my question](https://stats.stackexchange.com/questions/459639/is-there-any-problem-if-i-want-to-correlate-acceleration-with-calorie-regardless)? I have asked the question about 15 hours ago. However, still no answer received. – Md. Sabbir Ahmed Apr 11 '20 at 05:32
4

To add to Rob's answer ... with respect to reverse engineering a network, MI may be preferred over correlation when you want to extract causal rather than associative links in your network. Correlation networks are purely associative. But for MI, you need more data and computing power.

ars
  • 12,160
  • 1
  • 36
  • 54