Universal Approximation Power of Deep Residual Neural Networks via
  Nonlinear Control Theory
  In this paper, we explain the universal approximation capabilities of deep
residual neural networks through geometric nonlinear control. Inspired by
recent work establishing links between residual networks and control systems,
we provide a general sufficient condition for a residual network to have the
power of universal approximation by asking the activation function, or one of
its derivatives, to satisfy a quadratic differential equation. Many activation
functions used in practice satisfy this assumption, exactly or approximately,
and we show this property to be sufficient for an adequately deep neural
network with $n+1$ neurons per layer to approximate arbitrarily well, on a
compact set and with respect to the supremum norm, any continuous function from
$\mathbb{R}^n$ to $\mathbb{R}^n$. We further show this result to hold for very
simple architectures for which the weights only need to assume two values. The
first key technical contribution consists of relating the universal
approximation problem to controllability of an ensemble of control systems
corresponding to a residual network and to leverage classical Lie algebraic
techniques to characterize controllability. The second technical contribution
is to identify monotonicity as the bridge between controllability of finite
ensembles and uniform approximability on compact sets.
