Mean Embeddings with Test-Time Data Augmentation for Ensembling of
  Representations
  Averaging predictions over a set of models -- an ensemble -- is widely used
to improve predictive performance and uncertainty estimation of deep learning
models. At the same time, many machine learning systems, such as search,
matching, and recommendation systems, heavily rely on embeddings.
Unfortunately, due to misalignment of features of independently trained models,
embeddings, cannot be improved with a naive deep ensemble like approach. In
this work, we look at the ensembling of representations and propose mean
embeddings with test-time augmentation (MeTTA) simple yet well-performing
recipe for ensembling representations. Empirically we demonstrate that MeTTA
significantly boosts the quality of linear evaluation on ImageNet for both
supervised and self-supervised models. Even more exciting, we draw connections
between MeTTA, image retrieval, and transformation invariant models. We believe
that spreading the success of ensembles to inference higher-quality
representations is the important step that will open many new applications of
ensembling.
