Solving internal covariate shift in deep learning with linked neurons
  This work proposes a novel solution to the problem of internal covariate
shift and dying neurons using the concept of linked neurons. We define the
neuron linkage in terms of two constraints: first, all neuron activations in
the linkage must have the same operating point. That is to say, all of them
share input weights. Secondly, a set of neurons is linked if and only if there
is at least one member of the linkage that has a non-zero gradient in regard to
the input of the activation function. This means that for any input in the
activation function, there is at least one member of the linkage that operates
in a non-flat and non-zero area. This simple change has profound implications
in the network learning dynamics. In this article we explore the consequences
of this proposal and show that by using this kind of units, internal covariate
shift is implicitly solved. As a result of this, the use of linked neurons
allows to train arbitrarily large networks without any architectural or
algorithmic trick, effectively removing the need of using re-normalization
schemes such as Batch Normalization, which leads to halving the required
training time. It also solves the problem of the need for standarized input
data. Results show that the units using the linkage not only do effectively
solve the aforementioned problems, but are also a competitive alternative with
respect to state-of-the-art with very promising results.
