Geometric Regularization from Overparameterization explains Double
  Descent and other findings
  The volume of the distribution of possible weight configurations associated
with a loss value may be the source of implicit regularization from
overparameterization due to the phenomenon of contracting volume with
increasing dimensions for geometric figures demonstrated by hyperspheres. This
paper introduces geometric regularization and explores potential applicability
to several unexplained phenomenon including double descent, the differences
between wide and deep networks, the benefits of He initialization and retained
proximity in training, gradient confusion, fitness landscape properties, double
descent in other learning paradigms, and other findings for overparameterized
learning. Experiments are conducted by aggregating histograms of loss values
corresponding to randomly sampled initializations in small setups, which find
directional correlations in zero or central mode dominance from deviations in
width, depth, and initialization distributions. Double descent is likely due to
a regularization phase change when a training path reaches low enough loss that
the loss manifold volume contraction from a reduced range of potential weight
sets is amplified by an overparameterized geometry.
