Lipschitz regularity of deep neural networks: analysis and efficient
  estimation
  Deep neural networks are notorious for being sensitive to small well-chosen
perturbations, and estimating the regularity of such architectures is of utmost
importance for safe and robust practical applications. In this paper, we
investigate one of the key characteristics to assess the regularity of such
methods: the Lipschitz constant of deep learning architectures. First, we show
that, even for two layer neural networks, the exact computation of this
quantity is NP-hard and state-of-art methods may significantly overestimate it.
Then, we both extend and improve previous estimation methods by providing
AutoLip, the first generic algorithm for upper bounding the Lipschitz constant
of any automatically differentiable function. We provide a power method
algorithm working with automatic differentiation, allowing efficient
computations even on large convolutions. Second, for sequential neural
networks, we propose an improved algorithm named SeqLip that takes advantage of
the linear computation graph to split the computation per pair of consecutive
layers. Third we propose heuristics on SeqLip in order to tackle very large
networks. Our experiments show that SeqLip can significantly improve on the
existing upper bounds. Finally, we provide an implementation of AutoLip in the
PyTorch environment that may be used to better estimate the robustness of a
given neural network to small perturbations or regularize it using more precise
Lipschitz estimations.
