Statistical Neurodynamics of Deep Networks: Geometry of Signal Spaces
  Statistical neurodynamics studies macroscopic behaviors of randomly connected
neural networks. We consider a deep layered feedforward network where input
signals are processed layer by layer. The manifold of input signals is embedded
in a higher dimensional manifold of the next layer as a curved submanifold,
provided the number of neurons is larger than that of inputs. We show
geometrical features of the embedded manifold, proving that the manifold
enlarges or shrinks locally isotropically so that it is always embedded
conformally. We study the curvature of the embedded manifold. The scalar
curvature converges to a constant or diverges to infinity slowly. The distance
between two signals also changes, converging eventually to a stable fixed
value, provided both the number of neurons in a layer and the number of layers
tend to infinity. This causes a problem, since when we consider a curve in the
input space, it is mapped as a continuous curve of fractal nature, but our
theory contradictorily suggests that the curve eventually converges to a
discrete set of equally spaced points. In reality, the numbers of neurons and
layers are finite and thus, it is expected that the finite size effect causes
the discrepancies between our theory and reality. We need to further study the
discrepancies to understand their implications on information processing.
