Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse
  of Dimensionality on H\"older Class
  In this paper, we construct neural networks with ReLU, sine and $2^x$ as
activation functions. For general continuous $f$ defined on $[0,1]^d$ with
continuity modulus $\omega_f(\cdot)$, we construct ReLU-sine-$2^x$ networks
that enjoy an approximation rate
$\mathcal{O}(\omega_f(\sqrt{d})\cdot2^{-M}+\omega_{f}\left(\frac{\sqrt{d}}{N}\right))$,
where $M,N\in \mathbb{N}^{+}$ denote the hyperparameters related to widths of
the networks. As a consequence, we can construct ReLU-sine-$2^x$ network with
the depth $5$ and width
$\max\left\{\left\lceil2d^{3/2}\left(\frac{3\mu}{\epsilon}\right)^{1/{\alpha}}\right\rceil,2\left\lceil\log_2\frac{3\mu
d^{\alpha/2}}{2\epsilon}\right\rceil+2\right\}$ that approximates $f\in
\mathcal{H}_{\mu}^{\alpha}([0,1]^d)$ within a given tolerance $\epsilon >0$
measured in $L^p$ norm $p\in[1,\infty)$, where
$\mathcal{H}_{\mu}^{\alpha}([0,1]^d)$ denotes the H\"older continuous function
class defined on $[0,1]^d$ with order $\alpha \in (0,1]$ and constant $\mu >
0$. Therefore, the ReLU-sine-$2^x$ networks overcome the curse of
dimensionality on $\mathcal{H}_{\mu}^{\alpha}([0,1]^d)$. In addition to its
supper expressive power, functions implemented by ReLU-sine-$2^x$ networks are
(generalized) differentiable, enabling us to apply SGD to train.
