The universal approximation theorem for complex-valued neural networks
  We generalize the classical universal approximation theorem for neural
networks to the case of complex-valued neural networks. Precisely, we consider
feedforward networks with a complex activation function $\sigma : \mathbb{C}
\to \mathbb{C}$ in which each neuron performs the operation $\mathbb{C}^N \to
\mathbb{C}, z \mapsto \sigma(b + w^T z)$ with weights $w \in \mathbb{C}^N$ and
a bias $b \in \mathbb{C}$, and with $\sigma$ applied componentwise. We
completely characterize those activation functions $\sigma$ for which the
associated complex networks have the universal approximation property, meaning
that they can uniformly approximate any continuous function on any compact
subset of $\mathbb{C}^d$ arbitrarily well.
  Unlike the classical case of real networks, the set of "good activation
functions" which give rise to networks with the universal approximation
property differs significantly depending on whether one considers deep networks
or shallow networks: For deep networks with at least two hidden layers, the
universal approximation property holds as long as $\sigma$ is neither a
polynomial, a holomorphic function, or an antiholomorphic function. Shallow
networks, on the other hand, are universal if and only if the real part or the
imaginary part of $\sigma$ is not a polyharmonic function.
