Hierarchical Predictive Coding Models in a Deep-Learning Framework
  Bayesian predictive coding is a putative neuromorphic method for acquiring
higher-level neural representations to account for sensory input. Although
originating in the neuroscience community, there are also efforts in the
machine learning community to study these models. This paper reviews some of
the more well known models. Our review analyzes module connectivity and
patterns of information transfer, seeking to find general principles used
across the models. We also survey some recent attempts to cast these models
within a deep learning framework. A defining feature of Bayesian predictive
coding is that it uses top-down, reconstructive mechanisms to predict incoming
sensory inputs or their lower-level representations. Discrepancies between the
predicted and the actual inputs, known as prediction errors, then give rise to
future learning that refines and improves the predictive accuracy of learned
higher-level representations. Predictive coding models intended to describe
computations in the neocortex emerged prior to the development of deep learning
and used a communication structure between modules that we name the Rao-Ballard
protocol. This protocol was derived from a Bayesian generative model with some
rather strong statistical assumptions. The RB protocol provides a rubric to
assess the fidelity of deep learning models that claim to implement predictive
coding.
