MML: Maximal Multiverse Learning for Robust Fine-Tuning of Language
  Models
  Recent state-of-the-art language models utilize a two-phase training
procedure comprised of (i) unsupervised pre-training on unlabeled text, and
(ii) fine-tuning for a specific supervised task. More recently, many studies
have been focused on trying to improve these models by enhancing the
pre-training phase, either via better choice of hyperparameters or by
leveraging an improved formulation. However, the pre-training phase is
computationally expensive and often done on private datasets. In this work, we
present a method that leverages BERT's fine-tuning phase to its fullest, by
applying an extensive number of parallel classifier heads, which are enforced
to be orthogonal, while adaptively eliminating the weaker heads during
training. Our method allows the model to converge to an optimal number of
parallel classifiers, depending on the given dataset at hand.
  We conduct an extensive inter- and intra-dataset evaluations, showing that
our method improves the robustness of BERT, sometimes leading to a +9\% gain in
accuracy. These results highlight the importance of a proper fine-tuning
procedure, especially for relatively smaller-sized datasets. Our code is
attached as supplementary and our models will be made completely public.
