Any Target Function Exists in a Neighborhood of Any Sufficiently Wide
  Random Network: A Geometrical Perspective
  It is known that any target function is realized in a sufficiently small
neighborhood of any randomly connected deep network, provided the width (the
number of neurons in a layer) is sufficiently large. There are sophisticated
theories and discussions concerning this striking fact, but rigorous theories
are very complicated. We give an elementary geometrical proof by using a simple
model for the purpose of elucidating its structure. We show that
high-dimensional geometry plays a magical role: When we project a
high-dimensional sphere of radius 1 to a low-dimensional subspace, the uniform
distribution over the sphere reduces to a Gaussian distribution of negligibly
small covariances.
