On Power Laws in Deep Ensembles
  Ensembles of deep neural networks are known to achieve state-of-the-art
performance in uncertainty estimation and lead to accuracy improvement. In this
work, we focus on a classification problem and investigate the behavior of both
non-calibrated and calibrated negative log-likelihood (CNLL) of a deep ensemble
as a function of the ensemble size and the member network size. We indicate the
conditions under which CNLL follows a power law w.r.t. ensemble size or member
network size, and analyze the dynamics of the parameters of the discovered
power laws. Our important practical finding is that one large network may
perform worse than an ensemble of several medium-size networks with the same
total number of parameters (we call this ensemble a memory split). Using the
detected power law-like dependencies, we can predict (1) the possible gain from
the ensembling of networks with given structure, (2) the optimal memory split
given a memory budget, based on a relatively small number of trained networks.
  We describe the memory split advantage effect in more details in
arXiv:2005.07292
