Characterizing Well-Behaved vs. Pathological Deep Neural Networks
  We introduce a novel approach, requiring only mild assumptions, for the
characterization of deep neural networks at initialization. Our approach
applies both to fully-connected and convolutional networks and easily
incorporates batch normalization and skip-connections. Our key insight is to
consider the evolution with depth of statistical moments of signal and noise,
thereby characterizing the presence or absence of pathologies in the hypothesis
space encoded by the choice of hyperparameters. We establish: (i) for
feedforward networks, with and without batch normalization, the
multiplicativity of layer composition inevitably leads to ill-behaved moments
and pathologies; (ii) for residual networks with batch normalization, on the
other hand, skip-connections induce power-law rather than exponential
behaviour, leading to well-behaved moments and no pathology.
