Convergence and Alignment of Gradient Descent with Random
  Backpropagation Weights
  Stochastic gradient descent with backpropagation is the workhorse of
artificial neural networks. It has long been recognized that backpropagation
fails to be a biologically plausible algorithm. Fundamentally, it is a
non-local procedure -- updating one neuron's synaptic weights requires
knowledge of synaptic weights or receptive fields of downstream neurons. This
limits the use of artificial neural networks as a tool for understanding the
biological principles of information processing in the brain. Lillicrap et al.
(2016) propose a more biologically plausible "feedback alignment" algorithm
that uses random and fixed backpropagation weights, and show promising
simulations. In this paper we study the mathematical properties of the feedback
alignment procedure by analyzing convergence and alignment for two-layer
networks under squared error loss. In the overparameterized setting, we prove
that the error converges to zero exponentially fast, and also that
regularization is necessary in order for the parameters to become aligned with
the random backpropagation weights. Simulations are given that are consistent
with this analysis and suggest further generalizations. These results
contribute to our understanding of how biologically plausible algorithms might
carry out weight learning in a manner different from Hebbian learning, with
performance that is comparable with the full non-local backpropagation
algorithm.
