Generalization Comparison of Deep Neural Networks via Output Sensitivity
  Although recent works have brought some insights into the performance
improvement of techniques used in state-of-the-art deep-learning models, more
work is needed to understand their generalization properties. We shed light on
this matter by linking the loss function to the output's sensitivity to its
input. We find a rather strong empirical relation between the output
sensitivity and the variance in the bias-variance decomposition of the loss
function, which hints on using sensitivity as a metric for comparing the
generalization performance of networks, without requiring labeled data. We find
that sensitivity is decreased by applying popular methods which improve the
generalization performance of the model, such as (1) using a deep network
rather than a wide one, (2) adding convolutional layers to baseline classifiers
instead of adding fully-connected layers, (3) using batch normalization,
dropout and max-pooling, and (4) applying parameter initialization techniques.
