From complex to simple : hierarchical free-energy landscape renormalized
  in deep neural networks
  We develop a statistical mechanical approach based on the replica method to
study the design space of deep and wide neural networks constrained to meet a
large number of training data. Specifically, we analyze the configuration space
of the synaptic weights and neurons in the hidden layers in a simple
feed-forward perceptron network for two scenarios: a setting with random
inputs/outputs and a teacher-student setting. By increasing the strength of
constraints,~i.e. increasing the number of training data, successive 2nd order
glass transition (random inputs/outputs) or 2nd order crystalline transition
(teacher-student setting) take place layer-by-layer starting next to the
inputs/outputs boundaries going deeper into the bulk with the thickness of the
solid phase growing logarithmically with the data size. This implies the
typical storage capacity of the network grows exponentially fast with the
depth. In a deep enough network, the central part remains in the liquid phase.
We argue that in systems of finite width N, the weak bias field can remain in
the center and plays the role of a symmetry-breaking field that connects the
opposite sides of the system. The successive glass transitions bring about a
hierarchical free-energy landscape with ultrametricity, which evolves in space:
it is most complex close to the boundaries but becomes renormalized into
progressively simpler ones in deeper layers. These observations provide clues
to understand why deep neural networks operate efficiently. Finally, we present
some numerical simulations of learning which reveal spatially heterogeneous
glassy dynamics truncated by a finite width $N$ effect.
