Analyzing noise in autoencoders and deep networks
  Autoencoders have emerged as a useful framework for unsupervised learning of
internal representations, and a wide variety of apparently conceptually
disparate regularization techniques have been proposed to generate useful
features. Here we extend existing denoising autoencoders to additionally inject
noise before the nonlinearity, and at the hidden unit activations. We show that
a wide variety of previous methods, including denoising, contractive, and
sparse autoencoders, as well as dropout can be interpreted using this
framework. This noise injection framework reaps practical benefits by providing
a unified strategy to develop new internal representations by designing the
nature of the injected noise. We show that noisy autoencoders outperform
denoising autoencoders at the very task of denoising, and are competitive with
other single-layer techniques on MNIST, and CIFAR-10. We also show that types
of noise other than dropout improve performance in a deep network through
sparsifying, decorrelating, and spreading information across representations.
