Intrinsic Dimension, Persistent Homology and Generalization in Neural
  Networks
  Disobeying the classical wisdom of statistical learning theory, modern deep
neural networks generalize well even though they typically contain millions of
parameters. Recently, it has been shown that the trajectories of iterative
optimization algorithms can possess fractal structures, and their
generalization error can be formally linked to the complexity of such fractals.
This complexity is measured by the fractal's intrinsic dimension, a quantity
usually much smaller than the number of parameters in the network. Even though
this perspective provides an explanation for why overparametrized networks
would not overfit, computing the intrinsic dimension (e.g., for monitoring
generalization during training) is a notoriously difficult task, where existing
methods typically fail even in moderate ambient dimensions. In this study, we
consider this problem from the lens of topological data analysis (TDA) and
develop a generic computational tool that is built on rigorous mathematical
foundations. By making a novel connection between learning theory and TDA, we
first illustrate that the generalization error can be equivalently bounded in
terms of a notion called the 'persistent homology dimension' (PHD), where,
compared with prior work, our approach does not require any additional
geometrical or statistical assumptions on the training dynamics. Then, by
utilizing recently established theoretical results and TDA tools, we develop an
efficient algorithm to estimate PHD in the scale of modern deep neural networks
and further provide visualization tools to help understand generalization in
deep learning. Our experiments show that the proposed approach can efficiently
compute a network's intrinsic dimension in a variety of settings, which is
predictive of the generalization error.
