Vanishing Nodes: Another Phenomenon That Makes Training Deep Neural
  Networks Difficult
  It is well known that the problem of vanishing/exploding gradients is a
challenge when training deep networks. In this paper, we describe another
phenomenon, called vanishing nodes, that also increases the difficulty of
training deep neural networks. As the depth of a neural network increases, the
network's hidden nodes have more highly correlated behavior. This results in
great similarities between these nodes. The redundancy of hidden nodes thus
increases as the network becomes deeper. We call this problem vanishing nodes,
and we propose the metric vanishing node indicator (VNI) for quantitatively
measuring the degree of vanishing nodes. The VNI can be characterized by the
network parameters, which is shown analytically to be proportional to the depth
of the network and inversely proportional to the network width. The theoretical
results show that the effective number of nodes vanishes to one when the VNI
increases to one (its maximal value), and that vanishing/exploding gradients
and vanishing nodes are two different challenges that increase the difficulty
of training deep neural networks. The numerical results from the experiments
suggest that the degree of vanishing nodes will become more evident during
back-propagation training, and that when the VNI is equal to 1, the network
cannot learn simple tasks (e.g. the XOR problem) even when the gradients are
neither vanishing nor exploding. We refer to this kind of gradients as the
walking dead gradients, which cannot help the network converge when having a
relatively large enough scale. Finally, the experiments show that the
likelihood of failed training increases as the depth of the network increases.
The training will become much more difficult due to the lack of network
representation capability.
