Probabilistic bounds on neuron death in deep rectifier networks
  Neuron death is a complex phenomenon with implications for model
trainability: the deeper the network, the lower the probability of finding a
valid initialization. In this work, we derive both upper and lower bounds on
the probability that a ReLU network is initialized to a trainable point, as a
function of model hyperparameters. We show that it is possible to increase the
depth of a network indefinitely, so long as the width increases as well.
Furthermore, our bounds are asymptotically tight under reasonable assumptions:
first, the upper bound coincides with the true probability for a single-layer
network with the largest possible input set. Second, the true probability
converges to our lower bound as the input set shrinks to a single point, or as
the network complexity grows under an assumption about the output variance. We
confirm these results by numerical simulation, showing rapid convergence to the
lower bound with increasing network depth. Then, motivated by the theory, we
propose a practical sign flipping scheme which guarantees that the ratio of
living data points in a $k$-layer network is at least $2^{-k}$. Finally, we
show how these issues are mitigated by network design features currently seen
in practice, such as batch normalization, residual connections, dense networks
and skip connections. This suggests that neuron death may provide insight into
the efficacy of various model architectures.
