Spontaneous Symmetry Breaking in Neural Networks
  We propose a framework to understand the unprecedented performance and
robustness of deep neural networks using field theory. Correlations between the
weights within the same layer can be described by symmetries in that layer, and
networks generalize better if such symmetries are broken to reduce the
redundancies of the weights. Using a two parameter field theory, we find that
the network can break such symmetries itself towards the end of training in a
process commonly known in physics as spontaneous symmetry breaking. This
corresponds to a network generalizing itself without any user input layers to
break the symmetry, but by communication with adjacent layers. In the layer
decoupling limit applicable to residual networks (He et al., 2015), we show
that the remnant symmetries that survive the non-linear layers are
spontaneously broken. The Lagrangian for the non-linear and weight layers
together has striking similarities with the one in quantum field theory of a
scalar. Using results from quantum field theory we show that our framework is
able to explain many experimentally observed phenomena,such as training on
random labels with zero error (Zhang et al., 2017), the information bottleneck,
the phase transition out of it and gradient variance explosion (Shwartz-Ziv &
Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more.
