Quasi-Equivalence of Width and Depth of Neural Networks
  While classic studies proved that wide networks allow universal
approximation, recent research and successes of deep learning demonstrate the
power of deep networks. Based on a symmetric consideration, we investigate if
the design of artificial neural networks should have a directional preference,
and what the mechanism of interaction is between the width and depth of a
network. Inspired by the De Morgan law, we address this fundamental question by
establishing a quasi-equivalence between the width and depth of ReLU networks
in two aspects. First, we formulate two transforms for mapping an arbitrary
ReLU network to a wide network and a deep network respectively for either
regression or classification so that the essentially same capability of the
original network can be implemented. Then, we replace the mainstream artificial
neuron type with a quadratic counterpart, and utilize the factorization and
continued fraction representations of the same polynomial function to construct
a wide network and a deep network, respectively. Based on our findings, a deep
network has a wide equivalent, and vice versa, subject to an arbitrarily small
error.
