What training reveals about neural network complexity
  This work explores the Benevolent Training Hypothesis (BTH) which argues that
the complexity of the function a deep neural network (NN) is learning can be
deduced by its training dynamics. Our analysis provides evidence for BTH by
relating the NN's Lipschitz constant at different regions of the input space
with the behavior of the stochastic training procedure. We first observe that
the Lipschitz constant close to the training data affects various aspects of
the parameter trajectory, with more complex networks having a longer
trajectory, bigger variance, and often veering further from their
initialization. We then show that NNs whose 1st layer bias is trained more
steadily (i.e., slowly and with little variation) have bounded complexity even
in regions of the input space that are far from any training point. Finally, we
find that steady training with Dropout implies a training- and data-dependent
generalization bound that grows poly-logarithmically with the number of
parameters. Overall, our results support the intuition that good training
behavior can be a useful bias towards good generalization.
