Regularisation of Neural Networks by Enforcing Lipschitz Continuity
  We investigate the effect of explicitly enforcing the Lipschitz continuity of
neural networks with respect to their inputs. To this end, we provide a simple
technique for computing an upper bound to the Lipschitz constant---for multiple
$p$-norms---of a feed forward neural network composed of commonly used layer
types. Our technique is then used to formulate training a neural network with a
bounded Lipschitz constant as a constrained optimisation problem that can be
solved using projected stochastic gradient methods. Our evaluation study shows
that the performance of the resulting models exceeds that of models trained
with other common regularisers. We also provide evidence that the
hyperparameters are intuitive to tune, demonstrate how the choice of norm for
computing the Lipschitz constant impacts the resulting model, and show that the
performance gains provided by our method are particularly noticeable when only
a small amount of training data is available.
