MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks
  Recent strategies achieved ensembling "for free" by fitting concurrently
diverse subnetworks inside a single base network. The main idea during training
is that each subnetwork learns to classify only one of the multiple inputs
simultaneously provided. However, the question of how to best mix these
multiple inputs has not been studied so far. In this paper, we introduce MixMo,
a new generalized framework for learning multi-input multi-output deep
subnetworks. Our key motivation is to replace the suboptimal summing operation
hidden in previous approaches by a more appropriate mixing mechanism. For that
purpose, we draw inspiration from successful mixed sample data augmentations.
We show that binary mixing in features - particularly with rectangular patches
from CutMix - enhances results by making subnetworks stronger and more diverse.
We improve state of the art for image classification on CIFAR-100 and Tiny
ImageNet datasets. Our easy to implement models notably outperform data
augmented deep ensembles, without the inference and memory overheads. As we
operate in features and simply better leverage the expressiveness of large
networks, we open a new line of research complementary to previous works.
