Stochastic Function Norm Regularization of Deep Networks
  Deep neural networks have had an enormous impact on image analysis.
State-of-the-art training methods, based on weight decay and DropOut, result in
impressive performance when a very large training set is available. However,
they tend to have large problems overfitting to small data sets. Indeed, the
available regularization methods deal with the complexity of the network
function only indirectly. In this paper, we study the feasibility of directly
using the $L_2$ function norm for regularization. Two methods to integrate this
new regularization in the stochastic backpropagation are proposed. Moreover,
the convergence of these new algorithms is studied. We finally show that they
outperform the state-of-the-art methods in the low sample regime on benchmark
datasets (MNIST and CIFAR10). The obtained results demonstrate very clear
improvement, especially in the context of small sample regimes with data laying
in a low dimensional manifold. Source code of the method can be found at
\url{https://github.com/AmalRT/DNN_Reg}.
