Arbitrary-Depth Universal Approximation Theorems for Operator Neural
  Networks
  The standard Universal Approximation Theorem for operator neural networks
(NNs) holds for arbitrary width and bounded depth. Here, we prove that operator
NNs of bounded width and arbitrary depth are universal approximators for
continuous nonlinear operators. In our main result, we prove that for
non-polynomial activation functions that are continuously differentiable at a
point with a nonzero derivative, one can construct an operator NN of width
five, whose inputs are real numbers with finite decimal representations, that
is arbitrarily close to any given continuous nonlinear operator. We derive an
analogous result for non-affine polynomial activation functions. We also show
that depth has theoretical advantages by constructing operator ReLU NNs of
depth $2k^3+8$ and constant width that cannot be well-approximated by any
operator ReLU NN of depth $k$, unless its width is exponential in $k$.
