Skew Orthogonal Convolutions
  Training convolutional neural networks with a Lipschitz constraint under the
$l_{2}$ norm is useful for provable adversarial robustness, interpretable
gradients, stable training, etc. While 1-Lipschitz networks can be designed by
imposing a 1-Lipschitz constraint on each layer, training such networks
requires each layer to be gradient norm preserving (GNP) to prevent gradients
from vanishing. However, existing GNP convolutions suffer from slow training,
lead to significant reduction in accuracy and provide no guarantees on their
approximations. In this work, we propose a GNP convolution layer called Skew
Orthogonal Convolution (SOC) that uses the following mathematical property:
when a matrix is {\it Skew-Symmetric}, its exponential function is an {\it
orthogonal} matrix. To use this property, we first construct a convolution
filter whose Jacobian is Skew-Symmetric. Then, we use the Taylor series
expansion of the Jacobian exponential to construct the SOC layer that is
orthogonal. To efficiently implement SOC, we keep a finite number of terms from
the Taylor series and provide a provable guarantee on the approximation error.
Our experiments on CIFAR-10 and CIFAR-100 show that SOC allows us to train
provably Lipschitz, large convolutional neural networks significantly faster
than prior works while achieving significant improvements for both standard and
certified robust accuracies.
