Variational Autoencoders: A Harmonic Perspective
  In this work we study Variational Autoencoders (VAEs) from the perspective of
harmonic analysis. By viewing a VAE's latent space as a Gaussian Space, a
variety of measure space, we derive a series of results that show that the
encoder variance of a VAE controls the frequency content of the functions
parameterised by the VAE encoder and decoder neural networks. In particular we
demonstrate that larger encoder variances reduce the high frequency content of
these functions. Our analysis allows us to show that increasing this variance
effectively induces a soft Lipschitz constraint on the decoder network of a
VAE, which is a core contributor to the adversarial robustness of VAEs. We
further demonstrate that adding Gaussian noise to the input of a VAE allows us
to more finely control the frequency content and the Lipschitz constant of the
VAE encoder networks. To support our theoretical analysis we run experiments
with VAEs with small fully-connected neural networks and with larger
convolutional networks, demonstrating empirically that our theory holds for a
variety of neural network architectures.
