Deep Neural Networks with Short Circuits for Improved Gradient Learning
  Deep neural networks have achieved great success both in computer vision and
natural language processing tasks. However, mostly state-of-art methods highly
rely on external training or computing to improve the performance. To alleviate
the external reliance, we proposed a gradient enhancement approach, conducted
by the short circuit neural connections, to improve the gradient learning of
deep neural networks. The proposed short circuit is a unidirectional connection
that single back propagates the sensitive from the deep layer to the shallows.
Moreover, the short circuit formulates to be a gradient truncation of its
crossing layers which can plug into the backbone deep neural networks without
introducing external training parameters. Extensive experiments demonstrate
deep neural networks with our short circuit gain a large margin over the
baselines on both computer vision and natural language processing tasks.
