Extension of Direct Feedback Alignment to Convolutional and Recurrent
  Neural Network for Bio-plausible Deep Learning
  Throughout this paper, we focus on the improvement of the direct feedback
alignment (DFA) algorithm and extend the usage of the DFA to convolutional and
recurrent neural networks (CNNs and RNNs). Even though the DFA algorithm is
biologically plausible and has a potential of high-speed training, it has not
been considered as the substitute for back-propagation (BP) due to the low
accuracy in the CNN and RNN training. In this work, we propose a new DFA
algorithm for BP-level accurate CNN and RNN training. Firstly, we divide the
network into several modules and apply the DFA algorithm within the module.
Second, the DFA with the sparse backward weight is applied. It comes with a
form of dilated convolution in the CNN case, and in a form of sparse matrix
multiplication in the RNN case. Additionally, the error propagation method of
CNN becomes simpler through the group convolution. Finally, hybrid DFA
increases the accuracy of the CNN and RNN training to the BP-level while taking
advantage of the parallelism and hardware efficiency of the DFA algorithm.
