Information-Theoretic Local Minima Characterization and Regularization
  Recent advances in deep learning theory have evoked the study of
generalizability across different local minima of deep neural networks (DNNs).
While current work focused on either discovering properties of good local
minima or developing regularization techniques to induce good local minima, no
approach exists that can tackle both problems. We achieve these two goals
successfully in a unified manner. Specifically, based on the observed Fisher
information we propose a metric both strongly indicative of generalizability of
local minima and effectively applied as a practical regularizer. We provide
theoretical analysis including a generalization bound and empirically
demonstrate the success of our approach in both capturing and improving the
generalizability of DNNs. Experiments are performed on CIFAR-10, CIFAR-100 and
ImageNet for various network architectures.
