Skip Connections Eliminate Singularities
  Skip connections made the training of very deep networks possible and have
become an indispensable component in a variety of neural architectures. A
completely satisfactory explanation for their success remains elusive. Here, we
present a novel explanation for the benefits of skip connections in training
very deep networks. The difficulty of training deep networks is partly due to
the singularities caused by the non-identifiability of the model. Several such
singularities have been identified in previous works: (i) overlap singularities
caused by the permutation symmetry of nodes in a given layer, (ii) elimination
singularities corresponding to the elimination, i.e. consistent deactivation,
of nodes, (iii) singularities generated by the linear dependence of the nodes.
These singularities cause degenerate manifolds in the loss landscape that slow
down learning. We argue that skip connections eliminate these singularities by
breaking the permutation symmetry of nodes, by reducing the possibility of node
elimination and by making the nodes less linearly dependent. Moreover, for
typical initializations, skip connections move the network away from the
"ghosts" of these singularities and sculpt the landscape around them to
alleviate the learning slow-down. These hypotheses are supported by evidence
from simplified models, as well as from experiments with deep networks trained
on real-world datasets.
