Width is Less Important than Depth in ReLU Neural Networks
  We solve an open question from Lu et al. (2017), by showing that any target
network with inputs in $\mathbb{R}^d$ can be approximated by a width $O(d)$
network (independent of the target network's architecture), whose number of
parameters is essentially larger only by a linear factor. In light of previous
depth separation theorems, which imply that a similar result cannot hold when
the roles of width and depth are interchanged, it follows that depth plays a
more significant role than width in the expressive power of neural networks.
  We extend our results to constructing networks with bounded weights, and to
constructing networks with width at most $d+2$, which is close to the minimal
possible width due to previous lower bounds. Both of these constructions cause
an extra polynomial factor in the number of parameters over the target network.
We also show an exact representation of wide and shallow networks using deep
and narrow networks which, in certain cases, does not increase the number of
parameters over the target network.
