Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU
  Networks : Function Approximation and Statistical Recovery
  Real world data often exhibit low-dimensional geometric structures, and can
be viewed as samples near a low-dimensional manifold. This paper studies
nonparametric regression of H\"{o}lder functions on low-dimensional manifolds
using deep ReLU networks. Suppose $n$ training data are sampled from a
H\"{o}lder function in $\mathcal{H}^{s,\alpha}$ supported on a $d$-dimensional
Riemannian manifold isometrically embedded in $\mathbb{R}^D$, with sub-gaussian
noise. A deep ReLU network architecture is designed to estimate the underlying
function from the training data. The mean squared error of the empirical
estimator is proved to converge in the order of
$n^{-\frac{2(s+\alpha)}{2(s+\alpha) + d}}\log^3 n$. This result shows that deep
ReLU networks give rise to a fast convergence rate depending on the data
intrinsic dimension $d$, which is usually much smaller than the ambient
dimension $D$. It therefore demonstrates the adaptivity of deep ReLU networks
to low-dimensional geometric structures of data, and partially explains the
power of deep ReLU networks in tackling high-dimensional data with
low-dimensional geometric structures.
