Analytical bounds on the local Lipschitz constants of ReLU networks
  In this paper, we determine analytical upper bounds on the local Lipschitz
constants of feedforward neural networks with ReLU activation functions. We do
so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max
pooling functions, and combining the results to determine a network-wide bound.
Our method uses several insights to obtain tight bounds, such as keeping track
of the zero elements of each layer, and analyzing the composition of affine and
ReLU functions. Furthermore, we employ a careful computational approach which
allows us to apply our method to large networks such as AlexNet and VGG-16. We
present several examples using different networks, which show how our local
Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how
our method can be applied to provide adversarial bounds for classification
networks. These results show that our method produces the largest known bounds
on minimum adversarial perturbations for large networks such as AlexNet and
VGG-16.
