Representation Learning for Sequence Data with Deep Autoencoding
  Predictive Components
  We propose Deep Autoencoding Predictive Components (DAPC) -- a
self-supervised representation learning method for sequence data, based on the
intuition that useful representations of sequence data should exhibit a simple
structure in the latent space. We encourage this latent structure by maximizing
an estimate of predictive information of latent feature sequences, which is the
mutual information between past and future windows at each time step. In
contrast to the mutual information lower bound commonly used by contrastive
learning, the estimate of predictive information we adopt is exact under a
Gaussian assumption. Additionally, it can be computed without negative
sampling. To reduce the degeneracy of the latent space extracted by powerful
encoders and keep useful information from the inputs, we regularize predictive
information learning with a challenging masked reconstruction loss. We
demonstrate that our method recovers the latent space of noisy dynamical
systems, extracts predictive features for forecasting tasks, and improves
automatic speech recognition when used to pretrain the encoder on large amounts
of unlabeled data.
