Embedded Ensembles: Infinite Width Limit and Operating Regimes
  A memory efficient approach to ensembling neural networks is to share most
weights among the ensembled models by means of a single reference network. We
refer to this strategy as Embedded Ensembling (EE); its particular examples are
BatchEnsembles and Monte-Carlo dropout ensembles. In this paper we perform a
systematic theoretical and empirical analysis of embedded ensembles with
different number of models. Theoretically, we use a Neural-Tangent-Kernel-based
approach to derive the wide network limit of the gradient descent dynamics. In
this limit, we identify two ensemble regimes - independent and collective -
depending on the architecture and initialization strategy of ensemble models.
We prove that in the independent regime the embedded ensemble behaves as an
ensemble of independent models. We confirm our theoretical prediction with a
wide range of experiments with finite networks, and further study empirically
various effects such as transition between the two regimes, scaling of ensemble
performance with the network width and number of models, and dependence of
performance on a number of architecture and hyperparameter choices.
