Adding a vector of values to encoder output in autoencoder (keras)

Question

I am experimenting with autoencoders for a very specific application, but cannot unfortunately go into the specific details of what I am doing yet (fingers crossed I can do so after I make some progress).

Traditionally autoencoders have the encoder followed by the decoder. After the encoder, I add both a normalization and Gaussian noise step before adding the decoder as shown below. However, instead of adding Gaussian noise like in the example code, I want to add "non-random noise" for each sample. By "non-random noise", I mean augmenting the encoder with a vector of values. So for each data sample, I want to add this precomputed vector of values for that sample (the vector will differ by sample, so it's dependent on the sample itself) at the encoder level as noise.

I am not super familiar with deep learning frameworks, but is there any way to do this in keras and if so, how? If not, can I do it using tensor flow instead? I have included some of the code I have gotten started with:

def build_autoencoder(input_dim, encoded_dim, noise_std):
  autoencoder = Sequential()

  # Encoder Layers
  autoencoder.add(Dense(encoded_dim, input_shape=(input_dim,), activation='relu'))
  autoencoder.add(Dense(encoded_dim, activation='linear'))

  # add normalization layer
  autoencoder.add(Lambda(lambda x: K.l2_normalize(x,axis=1)))

  # add Gaussian noise
  # TODO: replace this with non-random noise for each sample
  autoencoder.add(GaussianNoise(noise_std))

  # Decoder layer
  autoencoder.add(Dense(input_dim, activation='sigmoid'))

  return autoencoder

Answer 1

Autoencoders take some data $\mathbf{x}$ and encode it into latent vector $\mathbf{z}$ of smaller dimensionality, to be able to reproduce $\mathbf{x}$ using the decoder:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) \to \mathbf{z} \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

where we train it by minimizing the loss, that measures how different is the reproduced $\mathbf{x}'$ from $\mathbf{x}$.

Variational autoencoders treat $\mathbf{z}$ as a random variable, so that we can sample from its distribution and generate the possible outputs of $\mathbf{x}$. Treating $\mathbf{z}$ as a random variable may additionally make the model more flexible, and will enable us to quantify the uncertainties about the estimates, but those are not the main reasons why people use VAEs. This is done by making the encoder to learn the parameters ($\boldsymbol{\mu}, \boldsymbol{\sigma}$) of the distribution of $\mathbf{z}$, rather then learning $\mathbf{z}$ directly:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) \to\; \mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma})\; \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

"Adding the random noise" is just a computational trick to achieving this.

If you drop the "random noise" from variational autoencoder, it becomes a (non-variational) autoencoder, just an "autoencoder". Same, if the noise would be deterministic, because in such case $\mathbf{z}$ wouldn't be a random variable any more, so you wouldn't be able to sample from it, and the main feature of the variational autoencoder would be lost.

Moreover, if you want to add "precomputed vector of values for that sample (the vector will differ by sample, so it's dependent on the sample itself)", then what you say is something like:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) + f(\mathbf{x}) \to \mathbf{z} \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

where $f(\mathbf{x})$ is some function of $\mathbf{x}$. Notice that this is just an autoencoder, that uses two encoder functions (sub-networks) and sums them, nothing more. Probably the same could be achieved with having single encoder, but with more parameters (neural networks are universal function approximators). If you mean that $f$ is going to have different structure then the $\mathsf{encoder}$, then this may make sense, but you would be using just an ensemble of different encoders as the encoder, where such approaches usually work better then the individual models.