Collegial Ensembles
  Modern neural network performance typically improves as model size increases.
A recent line of research on the Neural Tangent Kernel (NTK) of
over-parameterized networks indicates that the improvement with size increase
is a product of a better conditioned loss landscape. In this work, we
investigate a form of over-parameterization achieved through ensembling, where
we define collegial ensembles (CE) as the aggregation of multiple independent
models with identical architectures, trained as a single model. We show that
the optimization dynamics of CE simplify dramatically when the number of models
in the ensemble is large, resembling the dynamics of wide models, yet scale
much more favorably. We use recent theoretical results on the finite width
corrections of the NTK to perform efficient architecture search in a space of
finite width CE that aims to either minimize capacity, or maximize trainability
under a set of constraints. The resulting ensembles can be efficiently
implemented in practical architectures using group convolutions and block
diagonal layers. Finally, we show how our framework can be used to analytically
derive optimal group convolution modules originally found using expensive grid
searches, without having to train a single model.
