Complexity from Adaptive-Symmetries Breaking: Global Minima in the
  Statistical Mechanics of Deep Neural Networks
  An antithetical concept, adaptive symmetry, to conservative symmetry in
physics is proposed to understand the deep neural networks (DNNs). It
characterizes the invariance of variance, where a biotic system explores
different pathways of evolution with equal probability in absence of feedback
signals, and complex functional structure emerges from quantitative
accumulation of adaptive-symmetries breaking in response to feedback signals.
Theoretically and experimentally, we characterize the optimization process of a
DNN system as an extended adaptive-symmetry-breaking process. One particular
finding is that a hierarchically large DNN would have a large reservoir of
adaptive symmetries, and when the information capacity of the reservoir exceeds
the complexity of the dataset, the system could absorb all perturbations of the
examples and self-organize into a functional structure of zero training errors
measured by a certain surrogate risk. More specifically, this process is
characterized by a statistical-mechanical model that could be appreciated as a
generalization of statistics physics to the DNN organized complex system, and
characterizes regularities in higher dimensionality. The model consists of
three constitutes that could be appreciated as the counterparts of Boltzmann
distribution, Ising model, and conservative symmetry, respectively: (1) a
stochastic definition/interpretation of DNNs that is a multilayer probabilistic
graphical model, (2) a formalism of circuits that perform biological
computation, (3) a circuit symmetry from which self-similarity between the
microscopic and the macroscopic adaptability manifests. The model is analyzed
with a method referred as the statistical assembly method that analyzes the
coarse-grained behaviors (over a symmetry group) of the heterogeneous
hierarchical many-body interaction in DNNs.
