A random energy approach to deep learning
  We study a generic ensemble of deep belief networks which is parametrized by
the distribution of energy levels of the hidden states of each layer. We show
that, within a random energy approach, statistical dependence can propagate
from the visible to deep layers only if each layer is tuned close to the
critical point during learning. As a consequence, efficiently trained learning
machines are characterised by a broad distribution of energy levels. The
analysis of Deep Belief Networks and Restricted Boltzmann Machines on different
datasets confirms these conclusions.
