Fantastic Four: Differentiable Bounds on Singular Values of Convolution
  Layers
  In deep neural networks, the spectral norm of the Jacobian of a layer bounds
the factor by which the norm of a signal changes during forward/backward
propagation. Spectral norm regularizations have been shown to improve
generalization, robustness and optimization of deep learning methods. Existing
methods to compute the spectral norm of convolution layers either rely on
heuristics that are efficient in computation but lack guarantees or are
theoretically-sound but computationally expensive. In this work, we obtain the
best of both worlds by deriving {\it four} provable upper bounds on the
spectral norm of a standard 2D multi-channel convolution layer. These bounds
are differentiable and can be computed efficiently during training with
negligible overhead. One of these bounds is in fact the popular heuristic
method of Miyato et al. (multiplied by a constant factor depending on filter
sizes). Each of these four bounds can achieve the tightest gap depending on
convolution filters. Thus, we propose to use the minimum of these four bounds
as a tight, differentiable and efficient upper bound on the spectral norm of
convolution layers. We show that our spectral bound is an effective regularizer
and can be used to bound either the lipschitz constant or curvature values
(eigenvalues of the Hessian) of neural networks. Through experiments on MNIST
and CIFAR-10, we demonstrate the effectiveness of our spectral bound in
improving generalization and provable robustness of deep networks.
