1

I know KL divergence tries to measure how different 2 probability distributions are.

I know high correlation values between 2 sets of variables imply they are highly dependent on each other.

Will the probability distributions associated with both sets of variables have low KL divergence between them, i.e.: will they be similar?

gunes
  • 49,700
  • 3
  • 39
  • 75
unholy_me
  • 113
  • 4

1 Answers1

2

Not necessarily, e.g. $Y=cX$ and $X\sim N(0,1),\ c>0$, which means $Y\sim N(0,c^2)$. The KL divergence between two univariate normals can be calculated as laid out in here, and yields: $$KL(p_x||p_y)=2\log c+\frac{1}{2c^2}-{1\over2}$$

This can be arbitrarily large as $c$ changes but the correlation is always $1$.

gunes
  • 49,700
  • 3
  • 39
  • 75
  • Thank you for the explanation, however my naive intuition believed me to think that having high correlation implies that the data is sampled from similar distributions. – unholy_me Apr 09 '20 at 16:33