Approximation of Smoothness Classes by Deep Rectifier Networks
  We consider approximation rates of sparsely connected deep rectified linear
unit (ReLU) and rectified power unit (RePU) neural networks for functions in
Besov spaces $B^\alpha_{q}(L^p)$ in arbitrary dimension $d$, on general
domains. We show that \alert{deep rectifier} networks with a fixed activation
function attain optimal or near to optimal approximation rates for functions in
the Besov space $B^\alpha_{\tau}(L^\tau)$ on the critical embedding line
$1/\tau=\alpha/d+1/p$ for \emph{arbitrary} smoothness order $\alpha>0$. Using
interpolation theory, this implies that the entire range of smoothness classes
at or above the critical line is (near to) optimally approximated by deep
ReLU/RePU networks.
