Reverse Differentiation via Predictive Coding
  Deep learning has redefined the field of artificial intelligence (AI) thanks
to the rise of artificial neural networks, which are architectures inspired by
their neurological counterpart in the brain. Through the years, this dualism
between AI and neuroscience has brought immense benefits to both fields,
allowing neural networks to be used in dozens of applications. These networks
use an efficient implementation of reverse differentiation, called
backpropagation (BP). This algorithm, however, is often criticized for its
biological implausibility (e.g., lack of local update rules for the
parameters). Therefore, biologically plausible learning methods that rely on
predictive coding (PC), a framework for describing information processing in
the brain, are increasingly studied. Recent works prove that these methods can
approximate BP up to a certain margin on multilayer perceptrons (MLPs), and
asymptotically on any other complex model, and that zero-divergence inference
learning (Z-IL), a variant of PC, is able to exactly implement BP on MLPs.
However, the recent literature shows also that there is no biologically
plausible method yet that can exactly replicate the weight update of BP on
complex models. To fill this gap, in this paper, we generalize (PC and) Z-IL by
directly defining them on computational graphs, and show that it can perform
exact reverse differentiation. What results is the first biologically plausible
algorithm that is equivalent to BP in the way of updating parameters on any
neural network, providing a bridge between the interdisciplinary research of
neuroscience and deep learning.
