Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [variational-bayes]

Variational Bayesian methods approximate intractable integrals found in Bayesian inference and machine learning. Primarily, these methods serve one of two purposes: Approximating the posterior distribution, or bounding the marginal likelihood of observed data.

Variational Bayesian methods provide a locally optimal, exactly analytical solution to intractable integrals in Bayesian inference and machine learning. They are often presented as a faster, approximate alternative to MCMC methods of posterior inference.

342 questions
101
votes
7 answers

How does the reparameterization trick for VAEs work and why is it important?

How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
David Dao
  • 2,474
  • 3
  • 12
  • 16
46
votes
1 answer

Variational inference versus MCMC: when to choose one over the other?

I think I get the general idea of both VI and MCMC including the various flavors of MCMC like Gibbs sampling, Metropolis Hastings etc. This paper provides a wonderful exposition of both methods. I have the following questions: If I wish to do…
kedarps
  • 2,902
  • 2
  • 19
  • 30
45
votes
4 answers

how to weight KLD loss vs reconstruction loss in variational auto-encoder

in nearly all code examples I've seen of a VAE, the loss functions are defined as follows (this is tensorflow code, but I've seen similar for theano, torch etc. It's also for a convnet, but that's also not too relevant, just affects the axes the…
memo
  • 859
  • 1
  • 7
  • 10
40
votes
4 answers

When should I use a variational autoencoder as opposed to an autoencoder?

I understand the basic structure of variational autoencoder and normal (deterministic) autoencoder and the math behind them, but when and why would I prefer one type of autoencoder to the other? All I can think about is the prior distribution of…
DiveIntoML
  • 1,583
  • 1
  • 11
  • 21
34
votes
1 answer

Relation between variational Bayes and EM

I read somewhere that Variational Bayes method is a generalization of the EM algorithm. Indeed, the iterative parts of the algorithms are very similar. In order to test whether the EM algorithm is a special version of the Variational Bayes, I tried…
Ufuk Can Bicici
  • 2,028
  • 1
  • 17
  • 26
31
votes
1 answer

What are variational autoencoders and to what learning tasks are they used?

As per this and this answer, autoencoders seem to be a technique that uses neural networks for dimension reduction. I would like to additionally know what is a variational autoencoder (its main differences/benefits over a "traditional" autoencoders)…
user188529
30
votes
2 answers

What is the "capacity" of a machine learning model?

I'm studying this Tutorial on Variational Autoencoders by Carl Doersch. In the second page it states: One of the most popular such frameworks is the Variational Autoencoder [1, 3], the subject of this tutorial. The assumptions of this model are…
Andrés Marafioti
  • 513
  • 1
  • 4
  • 5
21
votes
3 answers

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

When having real valued entries (e.g. floats between 0 and 1 as normalized representation for greyscale values from 0 to 256) in our label vector, I always thought that we use MSE(R2-loss) if we want to measure the distance/error between input and…
SolingerMuc
  • 674
  • 1
  • 4
  • 12
20
votes
1 answer

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, the latent vector $\mathbf{z}$ is then calculated…
YellowPillow
  • 1,031
  • 2
  • 9
  • 16
15
votes
3 answers

What does a 'tractable' distribution mean?

For example, in generative adversarial network, we often hear that inference is easy because the conditional distribution of x given latent variable z is 'tractable'. Also, I read somewhere that Boltzmann machine and variational autoencoder is used…
sirius27
  • 161
  • 1
  • 3
14
votes
2 answers

Why do we use Gaussian distributions in Variational Autoencoder?

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and not another one ? This is maybe linked with…
Tbertin
  • 327
  • 2
  • 12
13
votes
1 answer

Variational Inference, KL divergence requires true $p$

To my (very modest) understand of variational inference, one tries to approximate an unknown distribution $p$ by finding a distribution $q$ that optimises the following: $$KL (p||q) = \sum\limits_{x} p(x)log \frac {p(x)}{q(x)}$$ Whenever I invest…
Vincent Warmerdam
  • 1,129
  • 1
  • 9
  • 10
13
votes
3 answers

what does one mean by numerical integration is too expensive?

I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" I do not have a background in mathematics and I was wondering what exactly does expensive mean here? Is it just…
discretetimeisnice
  • 427
  • 3
  • 12
12
votes
1 answer

How should I intuitively understand the KL divergence loss in variational autoencoders?

I was studying VAEs and came across the loss function that consists of the KL divergence. $$ \sum_{i=1}^n \sigma^2_i + \mu_i^2 - \log(\sigma_i) - 1 $$ I wanted to intuitively make sense of the KL divergence part of the loss function. It would be…
raptorAcrylyc
  • 125
  • 1
  • 1
  • 8
12
votes
1 answer

Comparing Laplace Approximation and Variational Inference

Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something like conditions on the distribution being…
aleshing
  • 1,418
  • 1
  • 8
  • 21
1
2 3
…
22 23