Relaxing Equivariance Constraints with Non-stationary Continuous Filters
  Equivariances provide useful inductive biases in neural network modeling,
with the translation equivariance of convolutional neural networks being a
canonical example. Equivariances can be embedded in architectures through
weight-sharing and place symmetry constraints on the functions a neural network
can represent. The type of symmetry is typically fixed and has to be chosen in
advance. Although some tasks are inherently equivariant, many tasks do not
strictly follow such symmetries. In such cases, equivariance constraints can be
overly restrictive. In this work, we propose a parameter-efficient relaxation
of equivariance that can effectively interpolate between a (i) non-equivariant
linear product, (ii) a strict-equivariant convolution, and (iii) a
strictly-invariant mapping. The proposed parameterization can be thought of as
a building block to allow adjustable symmetry structure in neural networks.
Compared to non-equivariant or strict-equivariant baselines, we experimentally
verify that soft equivariance leads to improved performance in terms of test
accuracy on CIFAR-10 and CIFAR-100 image classification tasks.
