Neural Networks Should Be Wide Enough to Learn Disconnected Decision
  Regions
  In the recent literature the important role of depth in deep learning has
been emphasized. In this paper we argue that sufficient width of a feedforward
network is equally important by answering the simple question under which
conditions the decision regions of a neural network are connected. It turns out
that for a class of activation functions including leaky ReLU, neural networks
having a pyramidal structure, that is no layer has more hidden units than the
input dimension, produce necessarily connected decision regions. This implies
that a sufficiently wide hidden layer is necessary to guarantee that the
network can produce disconnected decision regions. We discuss the implications
of this result for the construction of neural networks, in particular the
relation to the problem of adversarial manipulation of classifiers.
