7

When working with GAN's, is there a clear theoretical difference of choosing my latent distribution to be a uniform vs a normal distribution?

My intuition says that if I learn from a $z\sim N(\mu,\sigma)$ latent variable that has a small $\sigma$, that would mean that the samples my generator would produce would be very sparse and similar to each other; as if I were spanning a smaller space of Generated samples for my generator function $G(z)$ and the discriminator $D(x)$ would have a harder time discriminating between real and fake data if $G(z)$ is very close to $x \sim p_{data}(x)$.

On the other hand if $z\sim U[0,1]$, then perhaps the variability is less likely, but I believe that with an equally likelihood of exploring all the $[0,1]$ space then the Generator has a higher exploration space if viewing this from an exploration vs exploitation framework (as $G(z)$ is equally likely to hit all locations in the Generator space, vs a subset).

My thoughts here are somewhat hand-wavy and I'm looking for a paper, reference or proof that would confirm or reject what I'm thinking about.

Mecasickle
  • 292
  • 1
  • 12

2 Answers2

4

A sufficiently powerful function approximator can map any probability distribution to any other probability distribution. In practice, this means that any for reasonable choice of latent distribution, you can train a generator to map that to the distribution of images in your dataset.

So it doesn't make any fundamental difference to use uniform distribution over gaussian or vice versa. However, in variational autoencoders, where you have the encoder trying to predict the latent representation of the image $q(z|X)$, then a normal distribution makes things easier to work with, as your predictions can't ever go "out of bounds" as they could if you used a uniform distribution. Of course this doesn't apply to GANs.

The variability of the samples generated which you mentioned in your question isn't really a function of the latent distribution, since the network can scale up and down the spread of the distribution with a single layer.

shimao
  • 22,706
  • 2
  • 42
  • 81
  • What do you mean by out of bounds exactly? – thigi Jul 23 '18 at 15:39
  • @thigi $2$ would be out of bounds in a uniform distribution from $0$ to $1$. A gaussian has support everywhere so this is not an issue. – shimao Jul 23 '18 at 15:47
  • Yes but in your comment you said: a gaussian does not ever go out of bounds. here you basically say: 2 is out of bounds which can be produced by a gaussian but not by a uniform distribution. so what you wanted to say is: a gaussian could handle the "out of bounds" case. – thigi Jul 23 '18 at 17:05
  • Ah I see! I read it the wrong way! sorry I think you said it correctly. – thigi Jul 23 '18 at 17:06
1

I partially disagree with what @shimano said

The variability of the samples generated which you mentioned in your question isn't really a function of the latent distribution

The sampling space is very crucial for the GANs results. For instance, sampling $z\sim\mathbb{N}(\mu, \sigma)$ where $\sigma=1$ or $\sigma=10$ would ends up quite differently even when you dataset is not natural images (i.e. MNIST). The intuition behind this is where your prior ($z$) has very small STD it would create very similar results but with relatively high quality and on the other hand too large sigma would tend to create more versatile images but with relatively low quality. I am sorry that I don't have any reference for this statement but I explored a bit this area of prior sampling so it isn't only a gut feeling.

Idan Azuri
  • 111
  • 3