Activation Relaxation: A Local Dynamical Approximation to
  Backpropagation in the Brain
  The backpropagation of error algorithm (backprop) has been instrumental in
the recent success of deep learning. However, a key question remains as to
whether backprop can be formulated in a manner suitable for implementation in
neural circuitry. The primary challenge is to ensure that any candidate
formulation uses only local information, rather than relying on global signals
as in standard backprop. Recently several algorithms for approximating backprop
using only local signals have been proposed. However, these algorithms
typically impose other requirements which challenge biological plausibility:
for example, requiring complex and precise connectivity schemes, or multiple
sequential backwards phases with information being stored across phases. Here,
we propose a novel algorithm, Activation Relaxation (AR), which is motivated by
constructing the backpropagation gradient as the equilibrium point of a
dynamical system. Our algorithm converges rapidly and robustly to the correct
backpropagation gradients, requires only a single type of computational unit,
utilises only a single parallel backwards relaxation phase, and can operate on
arbitrary computation graphs. We illustrate these properties by training deep
neural networks on visual classification tasks, and describe simplifications to
the algorithm which remove further obstacles to neurobiological implementation
(for example, the weight-transport problem, and the use of nonlinear
derivatives), while preserving performance.
