Target-Embedding Autoencoders for Supervised Representation Learning
  Autoencoder-based learning has emerged as a staple for disciplining
representations in unsupervised and semi-supervised settings. This paper
analyzes a framework for improving generalization in a purely supervised
setting, where the target space is high-dimensional. We motivate and formalize
the general framework of target-embedding autoencoders (TEA) for supervised
prediction, learning intermediate latent representations jointly optimized to
be both predictable from features as well as predictive of targets---encoding
the prior that variations in targets are driven by a compact set of underlying
factors. As our theoretical contribution, we provide a guarantee of
generalization for linear TEAs by demonstrating uniform stability, interpreting
the benefit of the auxiliary reconstruction task as a form of regularization.
As our empirical contribution, we extend validation of this approach beyond
existing static classification applications to multivariate sequence
forecasting, verifying their advantage on both linear and nonlinear recurrent
architectures---thereby underscoring the further generality of this framework
beyond feedforward instantiations.
