Using Inherent Structures to design Lean 2-layer RBMs
  Understanding the representational power of Restricted Boltzmann Machines
(RBMs) with multiple layers is an ill-understood problem and is an area of
active research. Motivated from the approach of \emph{Inherent Structure
formalism} (Stillinger & Weber, 1982), extensively used in analysing Spin
Glasses, we propose a novel measure called \emph{Inherent Structure Capacity}
(ISC), which characterizes the representation capacity of a fixed architecture
RBM by the expected number of modes of distributions emanating from the RBM
with parameters drawn from a prior distribution. Though ISC is intractable, we
show that for a single layer RBM architecture ISC approaches a finite constant
as number of hidden units are increased and to further improve the ISC, one
needs to add a second layer. Furthermore, we introduce \emph{Lean} RBMs, which
are multi-layer RBMs where each layer can have at-most $O(n)$ units with the
number of visible units being n. We show that for every single layer RBM with
$\Omega(n^{2+r}), r \ge 0$, hidden units there exists a two-layered \emph{lean}
RBM with $\Theta(n^2)$ parameters with the same ISC, establishing that 2 layer
RBMs can achieve the same representational power as single-layer RBMs but using
far fewer number of parameters. To the best of our knowledge, this is the first
result which quantitatively establishes the need for layering.
