Scaling up learning with GAIT-prop
  Backpropagation of error (BP) is a widely used and highly successful learning
algorithm. However, its reliance on non-local information in propagating error
gradients makes it seem an unlikely candidate for learning in the brain. In the
last decade, a number of investigations have been carried out focused upon
determining whether alternative more biologically plausible computations can be
used to approximate BP. This work builds on such a local learning algorithm -
Gradient Adjusted Incremental Target Propagation (GAIT-prop) - which has
recently been shown to approximate BP in a manner which appears biologically
plausible. This method constructs local, layer-wise weight update targets in
order to enable plausible credit assignment. However, in deep networks, the
local weight updates computed by GAIT-prop can deviate from BP for a number of
reasons. Here, we provide and test methods to overcome such sources of error.
In particular, we adaptively rescale the locally-computed errors and show that
this significantly increases the performance and stability of the GAIT-prop
algorithm when applied to the CIFAR-10 dataset.
