Orthogonalizing Convolutional Layers with the Cayley Transform
  Recent work has highlighted several advantages of enforcing orthogonality in
the weight layers of deep networks, such as maintaining the stability of
activations, preserving gradient norms, and enhancing adversarial robustness by
enforcing low Lipschitz constants. Although numerous methods exist for
enforcing the orthogonality of fully-connected layers, those for convolutional
layers are more heuristic in nature, often focusing on penalty methods or
limited classes of convolutions. In this work, we propose and evaluate an
alternative approach to directly parameterize convolutional layers that are
constrained to be orthogonal. Specifically, we propose to apply the Cayley
transform to a skew-symmetric convolution in the Fourier domain, so that the
inverse convolution needed by the Cayley transform can be computed efficiently.
We compare our method to previous Lipschitz-constrained and orthogonal
convolutional layers and show that it indeed preserves orthogonality to a high
degree even for large convolutions. Applied to the problem of certified
adversarial robustness, we show that networks incorporating the layer
outperform existing deterministic methods for certified defense against
$\ell_2$-norm-bounded adversaries, while scaling to larger architectures than
previously investigated. Code is available at
https://github.com/locuslab/orthogonal-convolutions.
