Efficient Representation of Low-Dimensional Manifolds using Deep
  Networks
  We consider the ability of deep neural networks to represent data that lies
near a low-dimensional manifold in a high-dimensional space. We show that deep
networks can efficiently extract the intrinsic, low-dimensional coordinates of
such data. We first show that the first two layers of a deep network can
exactly embed points lying on a monotonic chain, a special type of piecewise
linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably,
the network can do this using an almost optimal number of parameters. We also
show that this network projects nearby points onto the manifold and then embeds
them with little error. We then extend these results to more general manifolds.
