Pay attention to your loss: understanding misconceptions about
  1-Lipschitz neural networks
  Lipschitz constrained networks have gathered considerable attention in deep
learning community, with usages ranging from Wasserstein distance estimation to
the training of certifiably robust classifiers. However they remain commonly
considered as less accurate, and their properties in learning are still not
fully understood. In this paper we clarify the matter: when it comes to
classification 1-Lipschitz neural networks enjoy several advantages over their
unconstrained counterpart. First, we show that these networks are as accurate
as classical ones, and can fit arbitrarily difficult boundaries. Then, relying
on a robustness metric which reflects operational needs we characterize the
most robust classifier: the WGAN discriminator. Next, we show that 1-Lipschitz
neural networks generalize well under milder assumptions. Finally, we show that
hyper-parameters of the loss are crucial for controlling the
accuracy-robustness trade-off. We conclude that they exhibit appealing
properties to pave the way toward provably accurate, and provably robust neural
networks.
