A Quantitative Geometric Approach to Neural Network Smoothness
  Fast and precise Lipschitz constant estimation of neural networks is an
important task for deep learning. Researchers have recently found an intrinsic
trade-off between the accuracy and smoothness of neural networks, so training a
network with a loose Lipschitz constant estimation imposes a strong
regularization and can hurt the model accuracy significantly. In this work, we
provide a unified theoretical framework, a quantitative geometric approach, to
address the Lipschitz constant estimation. By adopting this framework, we can
immediately obtain several theoretical results, including the computational
hardness of Lipschitz constant estimation and its approximability. Furthermore,
the quantitative geometric perspective can also provide some insights into
recent empirical observations that techniques for one norm do not usually
transfer to another one.
  We also implement the algorithms induced from this quantitative geometric
approach in a tool GeoLIP. These algorithms are based on semidefinite
programming (SDP). Our empirical evaluation demonstrates that GeoLIP is more
scalable and precise than existing tools on Lipschitz constant estimation for
$\ell_\infty$-perturbations. Furthermore, we also show its intricate relations
with other recent SDP-based techniques, both theoretically and empirically. We
believe that this unified quantitative geometric perspective can bring new
insights and theoretical tools to the investigation of neural-network
smoothness and robustness.
