Why ReLU Units Sometimes Die: Analysis of Single-Unit Error
  Backpropagation in Neural Networks
  Recently, neural networks in machine learning use rectified linear units
(ReLUs) in early processing layers for better performance. Training these
structures sometimes results in "dying ReLU units" with near-zero outputs. We
first explore this condition via simulation using the CIFAR-10 dataset and
variants of two popular convolutive neural network architectures. Our
explorations show that the output activation probability Pr[y>0] is generally
less than 0.5 at system convergence for layers that do not employ skip
connections, and this activation probability tends to decrease as one
progresses from input layer to output layer. Employing a simplified model of a
single ReLU unit trained by a variant of error backpropagation, we then perform
a statistical convergence analysis to explore the model's evolutionary
behavior. Our analysis describes the potentially-slower convergence speeds of
dying ReLU units, and this issue can occur regardless of how the weights are
initialized.
