ResNet with one-neuron hidden layers is a Universal Approximator
  We demonstrate that a very deep ResNet with stacked modules with one neuron
per hidden layer and ReLU activation functions can uniformly approximate any
Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$.
Because of the identity mapping inherent to ResNets, our network has
alternating layers of dimension one and $d$. This stands in sharp contrast to
fully connected networks, which are not universal approximators if their width
is the input dimension $d$ [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our
result implies an increase in representational power for narrow deep networks
by the ResNet architecture.
