Deep Networks on Toroids: Removing Symmetries Reveals the Structure of
  Flat Regions in the Landscape Geometry
  We systematize the approach to the investigation of deep neural network
landscapes by basing it on the geometry of the space of implemented functions
rather than the space of parameters. Grouping classifiers into equivalence
classes, we develop a standardized parameterization in which all symmetries are
removed, resulting in a toroidal topology. On this space, we explore the error
landscape rather than the loss. This lets us derive a meaningful notion of the
flatness of minimizers and of the geodesic paths connecting them. Using
different optimization algorithms that sample minimizers with different
flatness we study the mode connectivity and other characteristics. Testing a
variety of state-of-the-art architectures and benchmark datasets, we confirm
the correlation between flatness and generalization performance; we further
show that in function space flatter minima are closer to each other and that
the barriers along the geodesics connecting them are small. We also find that
minimizers found by variants of gradient descent can be connected by zero-error
paths with a single bend. We observe similar qualitative results in neural
networks with binary weights and activations, providing one of the first
results concerning the connectivity in this setting. Our results hinge on
symmetry removal, and are in remarkable agreement with the rich phenomenology
described by some recent analytical studies performed on simple shallow models.
