Predictive Coding Can Do Exact Backpropagation on Convolutional and
  Recurrent Neural Networks
  Predictive coding networks (PCNs) are an influential model for information
processing in the brain. They have appealing theoretical interpretations and
offer a single mechanism that accounts for diverse perceptual phenomena of the
brain. On the other hand, backpropagation (BP) is commonly regarded to be the
most successful learning method in modern machine learning. Thus, it is
exciting that recent work formulates inference learning (IL) that trains PCNs
to approximate BP. However, there are several remaining critical issues: (i) IL
is an approximation to BP with unrealistic/non-trivial requirements, (ii) IL
approximates BP in single-step weight updates; whether it leads to the same
point as BP after the weight updates are conducted for more steps is unknown,
and (iii) IL is computationally significantly more costly than BP. To solve
these issues, a variant of IL that is strictly equivalent to BP in fully
connected networks has been proposed. In this work, we build on this result by
showing that it also holds for more complex architectures, namely,
convolutional neural networks and (many-to-one) recurrent neural networks. To
our knowledge, we are the first to show that a biologically plausible algorithm
is able to exactly replicate the accuracy of BP on such complex architectures,
bridging the existing gap between IL and BP, and setting an unprecedented
performance for PCNs, which can now be considered as efficient alternatives to
BP.
