0

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\bigg(\frac{Q(X)}{P(X)}}\bigg)$$

While in the the code here it says

kl_loss = - 0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)

The formula for KL-divergence looks completely different from what is in this line of code. Can somebody familiar with variational autoencoders help?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
MegaNightdude
  • 83
  • 1
  • 6
  • I have figured it out. The full derivation is posted here: https://math.stackexchange.com/questions/3113528/evaluating-integrals-in-the-paper-auto-encoding-variational-bayes – MegaNightdude May 07 '19 at 05:38

2 Answers2

2

Note that this implementation assumes Gaussian distributions, so this loss (as far as I understand it, it would be helpful if yo uactually used the conventional notation used in math instead of some code snippet) measures the KL divergence between gaussians. You can find the whole derivation here on stats.se:

https://stats.stackexchange.com/questions/7440/kl-divergence-between-two-univariate-gaussians

flawr
  • 563
  • 4
  • 15
2

It'll help to first rewrite the code in MathJax, viz. $$D_{KL}^\text{other}(P\Vert Q)=-\frac12\left(1+\ln\sigma^2-\bar{z}^2-\sigma^2\right).$$(Unfortunately, whoever wrote the code gave the false impression in the naming that z_log_sigma denotes $\ln\sigma$ rather than $\ln\sigma^2$.) With $\bar{z}=\mu_1-\mu_2$ this formula becomes$$D_{KL}^\text{other}(P\Vert Q)=-\frac12-\ln\sigma+\frac12(\mu_1-\mu_2)^2+\frac12\sigma^2.$$This is the $\sigma_1=\sigma,\,\sigma_2=1$ special case of the two-Gaussians KL divergence in @flawr's link.

J.G.
  • 420
  • 2
  • 10