Preventing Gradient Attenuation in Lipschitz Constrained Convolutional
  Networks
  Lipschitz constraints under L2 norm on deep neural networks are useful for
provable adversarial robustness bounds, stable training, and Wasserstein
distance estimation. While heuristic approaches such as the gradient penalty
have seen much practical success, it is challenging to achieve similar
practical performance while provably enforcing a Lipschitz constraint. In
principle, one can design Lipschitz constrained architectures using the
composition property of Lipschitz functions, but Anil et al. recently
identified a key obstacle to this approach: gradient norm attenuation. They
showed how to circumvent this problem in the case of fully connected networks
by designing each layer to be gradient norm preserving. We extend their
approach to train scalable, expressive, provably Lipschitz convolutional
networks. In particular, we present the Block Convolution Orthogonal
Parameterization (BCOP), an expressive parameterization of orthogonal
convolution operations. We show that even though the space of orthogonal
convolutions is disconnected, the largest connected component of BCOP with 2n
channels can represent arbitrary BCOP convolutions over n channels. Our BCOP
parameterization allows us to train large convolutional networks with provable
Lipschitz bounds. Empirically, we find that it is competitive with existing
approaches to provable adversarial robustness and Wasserstein distance
estimation.
