Feedback alignment in deep convolutional networks
  Ongoing studies have identified similarities between neural representations
in biological networks and in deep artificial neural networks. This has led to
renewed interest in developing analogies between the backpropagation learning
algorithm used to train artificial networks and the synaptic plasticity rules
operative in the brain. These efforts are challenged by biologically
implausible features of backpropagation, one of which is a reliance on
symmetric forward and backward synaptic weights. A number of methods have been
proposed that do not rely on weight symmetry but, thus far, these have failed
to scale to deep convolutional networks and complex data. We identify principal
obstacles to the scalability of such algorithms and introduce several
techniques to mitigate them. We demonstrate that a modification of the feedback
alignment method that enforces a weaker form of weight symmetry, one that
requires agreement of weight sign but not magnitude, can achieve performance
competitive with backpropagation. Our results complement those of Bartunov et
al. (2018) and Xiao et al. (2018b) and suggest that mechanisms that promote
alignment of feedforward and feedback weights are critical for learning in deep
networks.
