Autoencoding Slow Representations for Semi-supervised Data Efficient
  Regression
  The slowness principle is a concept inspired by the visual cortex of the
brain. It postulates that the underlying generative factors of a quickly
varying sensory signal change on a slower time scale. Unsupervised learning of
intermediate representations utilizing abundant unlabeled sensory data can be
leveraged to perform data-efficient supervised downstream regression. In this
paper, we propose a general formulation of slowness for unsupervised
representation learning adding a slowness regularization term to the estimate
lower bound of the beta-VAE to encourage temporal similarity in observation and
latent space. Within this framework we compare existing slowness regularization
terms such as the L1 and L2 loss used in existing end-to-end methods, the
SlowVAE and propose a new term based on Brownian motion. We empirically
evaluate these slowness regularization terms with respect to their downstream
task performance and data efficiency. We find that slow representations lead to
equal or better downstream task performance and data efficiency in different
experiment domains when compared to representations without slowness
regularization. Finally, we discuss how the Frechet Inception Distance (FID),
traditionally used to determine the generative capabilities of GANs, can serve
as a measure to predict the performance of pre-trained Autoencoder model in a
supervised downstream task and accelerate hyperparameter search.
