Trainability of ReLU networks and Data-dependent Initialization
  In this paper, we study the trainability of rectified linear unit (ReLU)
networks. A ReLU neuron is said to be dead if it only outputs a constant for
any input. Two death states of neurons are introduced; tentative and permanent
death. A network is then said to be trainable if the number of permanently dead
neurons is sufficiently small for a learning task. We refer to the probability
of a network being trainable as trainability. We show that a network being
trainable is a necessary condition for successful training and the trainability
serves as an upper bound of successful training rates. In order to quantify the
trainability, we study the probability distribution of the number of active
neurons at the initialization. In many applications, over-specified or
over-parameterized neural networks are successfully employed and shown to be
trained effectively. With the notion of trainability, we show that
over-parameterization is both a necessary and a sufficient condition for
minimizing the training loss. Furthermore, we propose a data-dependent
initialization method in the over-parameterized setting. Numerical examples are
provided to demonstrate the effectiveness of the method and our theoretical
findings.
