Universal Approximation Properties for ODENet and ResNet
  We prove a universal approximation property (UAP) for a class of ODENet and a
class of ResNet, which are used in many deep learning algorithms. The UAP can
be stated as follows. Let $n$ and $m$ be the dimension of input and output
data, and assume $m\leq n$. Then we show that ODENet width $n+m$ with any
non-polynomial continuous activation function can approximate any continuous
function on a compact subset on $\mathbb{R}^n$. We also show that ResNet has
the same property as the depth tends to infinity. Furthermore, we derive
explicitly the gradient of a loss function with respect to a certain tuning
variable. We use this to construct a learning algorithm for ODENet. To
demonstrate the usefulness of this algorithm, we apply it to a regression
problem, a binary classification, and a multinomial classification in MNIST.
