Deep, Skinny Neural Networks are not Universal Approximators
  In order to choose a neural network architecture that will be effective for a
particular modeling problem, one must understand the limitations imposed by
each of the potential options. These limitations are typically described in
terms of information theoretic bounds, or by comparing the relative complexity
needed to approximate example functions between different architectures. In
this paper, we examine the topological constraints that the architecture of a
neural network imposes on the level sets of all the functions that it is able
to approximate. This approach is novel for both the nature of the limitations
and the fact that they are independent of network depth for a broad family of
activation functions.
