Properties of the geometry of solutions and capacity of multi-layer
  neural networks with Rectified Linear Units activations
  Rectified Linear Units (ReLU) have become the main model for the neural units
in current deep learning systems. This choice has been originally suggested as
a way to compensate for the so called vanishing gradient problem which can
undercut stochastic gradient descent (SGD) learning in networks composed of
multiple layers. Here we provide analytical results on the effects of ReLUs on
the capacity and on the geometrical landscape of the solution space in
two-layer neural networks with either binary or real-valued weights. We study
the problem of storing an extensive number of random patterns and find that,
quite unexpectedly, the capacity of the network remains finite as the number of
neurons in the hidden layer increases, at odds with the case of threshold units
in which the capacity diverges. Possibly more important, a large deviation
approach allows us to find that the geometrical landscape of the solution space
has a peculiar structure: while the majority of solutions are close in distance
but still isolated, there exist rare regions of solutions which are much more
dense than the similar ones in the case of threshold units. These solutions are
robust to perturbations of the weights and can tolerate large perturbations of
the inputs. The analytical results are corroborated by numerical findings.
