A Scale Invariant Flatness Measure for Deep Network Minima
  It has been empirically observed that the flatness of minima obtained from
training deep networks seems to correlate with better generalization. However,
for deep networks with positively homogeneous activations, most measures of
sharpness/flatness are not invariant to rescaling of the network parameters,
corresponding to the same function. This means that the measure of
flatness/sharpness can be made as small or as large as possible through
rescaling, rendering the quantitative measures meaningless. In this paper we
show that for deep networks with positively homogenous activations, these
rescalings constitute equivalence relations, and that these equivalence
relations induce a quotient manifold structure in the parameter space. Using
this manifold structure and an appropriate metric, we propose a Hessian-based
measure for flatness that is invariant to rescaling. We use this new measure to
confirm the proposition that Large-Batch SGD minima are indeed sharper than
Small-Batch SGD minima.
