Learning Feature Hierarchies with Centered Deep Boltzmann Machines
  Deep Boltzmann machines are in principle powerful models for extracting the
hierarchical structure of data. Unfortunately, attempts to train layers jointly
(without greedy layer-wise pretraining) have been largely unsuccessful. We
propose a modification of the learning algorithm that initially recenters the
output of the activation functions to zero. This modification leads to a better
conditioned Hessian and thus makes learning easier. We test the algorithm on
real data and demonstrate that our suggestion, the centered deep Boltzmann
machine, learns a hierarchy of increasingly abstract representations and a
better generative model of data.
