Training Matters: Unlocking Potentials of Deeper Graph Convolutional
  Neural Networks
  The performance limit of Graph Convolutional Networks (GCNs) and the fact
that we cannot stack more of them to increase the performance, which we usually
do for other deep learning paradigms, are pervasively thought to be caused by
the limitations of the GCN layers, including insufficient expressive power,
etc. However, if so, for a fixed architecture, it would be unlikely to lower
the training difficulty and to improve performance by changing only the
training procedure, which we show in this paper not only possible but possible
in several ways. This paper first identify the training difficulty of GCNs from
the perspective of graph signal energy loss. More specifically, we find that
the loss of energy in the backward pass during training nullifies the learning
of the layers closer to the input. Then, we propose several methodologies to
mitigate the training problem by slightly modifying the GCN operator, from the
energy perspective. After empirical validation, we confirm that these changes
of operator lead to significant decrease in the training difficulties and
notable performance boost, without changing the composition of parameters. With
these, we conclude that the root cause of the problem is more likely the
training difficulty than the others.
