I'm trying to understand one specific formula in a paper that I'm reading:
https://arxiv.org/pdf/1911.02469.pdf
It's concerning equation 10:
Unfortunately, the authors don't explain the context of what is σ. I also tried to find other sources. In other implementations it seems quite common to just use conventional loss functions of the author's liking, without much explanation.
Recall VAE's loss has two components: reconstruction loss(since autoencoder's aim to learn to reconstruct) and KL loss (to measure how much information is lost or how much we have diverged from the prior). The actual form of the VAE loss(aim is to maximize this loss) is :
$$ L(\theta , \phi) = \sum_{i=1}^{N} E_{z_{i} \sim q_{\phi}(z|x_{i})} \left [ log p_{\theta} (x_{i}|z)\right] - KL(q_{\phi} (z | x_{i}) || p(z)) $$ where $\left (x , z \right)$ is input and latent vector pair. Encoder and decoder networks are $q$ and $p$ respectively. Since, we have a Gaussian prior, reconstruction loss becomes the squared difference(L2 distance) between input and reconstruction.(logarithm of gaussian reduces to squared difference).
In the paper, authors are trying to get intuition from probabilistic PCA to explain when the posterior collapse happens. pPCA model,trained EM or gradeint ascent on NLL. $logp(x)$, is defined as follows:
\begin{align} p(z) &= \textit{N}(0, I) \\ p(x|z) &= \textit{N}(Wz + \mu, \sigma ^{2} I) \end{align}
where $x$ and $z$ is the data and the latent respectively. The $\sigma ^{2}$ here is the variance of the observation noise. Posterior collapse turns out appers as stationary points of NLL. Authors show that $\sigma ^{2}$ affects the stability of the collapse stationary points in pPCA. Also, they show that this is similar in deep extension of pPCA (Deep Gaussian VAE).
So, the $\sigma ^{2}$ is derived from the first loss component of VAE while assuming that the data distribution has an inherent observation noise. You still use standard normal Gaussian prior as it is.
