Improving the Sample-Complexity of Deep Classification Networks with
  Invariant Integration
  Leveraging prior knowledge on intraclass variance due to transformations is a
powerful method to improve the sample complexity of deep neural networks. This
makes them applicable to practically important use-cases where training data is
scarce. Rather than being learned, this knowledge can be embedded by enforcing
invariance to those transformations. Invariance can be imposed using
group-equivariant convolutions followed by a pooling operation.
  For rotation-invariance, previous work investigated replacing the spatial
pooling operation with invariant integration which explicitly constructs
invariant representations. Invariant integration uses monomials which are
selected using an iterative approach requiring expensive pre-training. We
propose a novel monomial selection algorithm based on pruning methods to allow
an application to more complex problems. Additionally, we replace monomials
with different functions such as weighted sums, multi-layer perceptrons and
self-attention, thereby streamlining the training of
invariant-integration-based architectures.
  We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and
CIFAR-10 datasets where rotation-invariant-integration-based Wide-ResNet
architectures using monomials and weighted sums outperform the respective
baselines in the limited sample regime. We achieve state-of-the-art results
using full data on Rotated-MNIST and SVHN where rotation is a main source of
intraclass variation. On STL-10 we outperform a standard and a
rotation-equivariant convolutional neural network using pooling.
