Quantization-Based Regularization for Autoencoders
  Autoencoders and their variations provide unsupervised models for learning
low-dimensional representations for downstream tasks. Without proper
regularization, autoencoder models are susceptible to the overfitting problem
and the so-called posterior collapse phenomenon. In this paper, we introduce a
quantization-based regularizer in the bottleneck stage of autoencoder models to
learn meaningful latent representations. We combine both perspectives of Vector
Quantized-Variational AutoEncoders (VQ-VAE) and classical denoising
regularization methods of neural networks. We interpret quantizers as
regularizers that constrain latent representations while fostering a
similarity-preserving mapping at the encoder. Before quantization, we impose
noise on the latent codes and use a Bayesian estimator to optimize the
quantizer-based representation. The introduced bottleneck Bayesian estimator
outputs the posterior mean of the centroids to the decoder, and thus, is
performing soft quantization of the noisy latent codes. We show that our
proposed regularization method results in improved latent representations for
both supervised learning and clustering downstream tasks when compared to
autoencoders using other bottleneck structures.
