Ensemble of Averages: Improving Model Selection and Boosting Performance
  in Domain Generalization
  In Domain Generalization (DG) settings, models trained independently on a
given set of training domains have notoriously chaotic performance on
distribution shifted test domains, and stochasticity in optimization (e.g.
seed) plays a big role. This makes deep learning models unreliable in real
world settings. We first show that this chaotic behavior exists even along the
training optimization trajectory of a single model, and propose a simple model
averaging protocol that both significantly boosts domain generalization and
diminishes the impact of stochasticity by improving the rank correlation
between the in-domain validation accuracy and out-domain test accuracy, which
is crucial for reliable early stopping. Taking advantage of our observation, we
show that instead of ensembling unaveraged models (that is typical in
practice), ensembling moving average models (EoA) from independent runs further
boosts performance. We theoretically explain the boost in performance of
ensembling and model averaging by adapting the well known Bias-Variance
trade-off to the domain generalization setting. On the DomainBed benchmark,
when using a pre-trained ResNet-50, this ensemble of averages achieves an
average of $68.0\%$, beating vanilla ERM (w/o averaging/ensembling) by $\sim
4\%$, and when using a pre-trained RegNetY-16GF, achieves an average of
$76.6\%$, beating vanilla ERM by $6\%$. Our code is available at
\url{https://github.com/salesforce/ensemble-of-averages}.
