A Geometric Modeling of Occam's Razor in Deep Learning
  Why do deep neural networks (DNNs) benefit from very high dimensional
parameter spaces? Their huge parameter complexities vs stunning performances in
practice is all the more intriguing and not explainable using the standard
theory of regular models. In this work, we propose a geometrically flavored
information-theoretic approach to study this phenomenon. Namely, we introduce
the locally varying dimensionality of the parameter space of neural network
models by considering the number of significant dimensions of the Fisher
information matrix, and model the parameter space as a manifold using the
framework of singular semi-Riemannian geometry. We derive model complexity
measures which yield short description lengths for deep neural network models
based on their singularity analysis thus explaining the good performance of
DNNs despite their large number of parameters.
