Tangent Space Sensitivity and Distribution of Linear Regions in ReLU
  Networks
  Recent articles indicate that deep neural networks are efficient models for
various learning problems. However they are often highly sensitive to various
changes that cannot be detected by an independent observer. As our
understanding of deep neural networks with traditional generalization bounds
still remains incomplete, there are several measures which capture the
behaviour of the model in case of small changes at a specific state. In this
paper we consider adversarial stability in the tangent space and suggest
tangent sensitivity in order to characterize stability. We focus on a
particular kind of stability with respect to changes in parameters that are
induced by individual examples without known labels. We derive several easily
computable bounds and empirical measures for feed-forward fully connected ReLU
(Rectified Linear Unit) networks and connect tangent sensitivity to the
distribution of the activation regions in the input space realized by the
network. Our experiments suggest that even simple bounds and measures are
associated with the empirical generalization gap.
