Latent Equilibrium: A unified learning theory for arbitrarily fast
  computation with arbitrarily slow neurons
  The response time of physical computational elements is finite, and neurons
are no exception. In hierarchical models of cortical networks each layer thus
introduces a response lag. This inherent property of physical dynamical systems
results in delayed processing of stimuli and causes a timing mismatch between
network output and instructive signals, thus afflicting not only inference, but
also learning. We introduce Latent Equilibrium, a new framework for inference
and learning in networks of slow components which avoids these issues by
harnessing the ability of biological neurons to phase-advance their output with
respect to their membrane potential. This principle enables quasi-instantaneous
inference independent of network depth and avoids the need for phased
plasticity or computationally expensive network relaxation phases. We jointly
derive disentangled neuron and synapse dynamics from a prospective energy
function that depends on a network's generalized position and momentum. The
resulting model can be interpreted as a biologically plausible approximation of
error backpropagation in deep cortical networks with continuous-time, leaky
neuronal dynamics and continuously active, local plasticity. We demonstrate
successful learning of standard benchmark datasets, achieving competitive
performance using both fully-connected and convolutional architectures, and
show how our principle can be applied to detailed models of cortical
microcircuitry. Furthermore, we study the robustness of our model to
spatio-temporal substrate imperfections to demonstrate its feasibility for
physical realization, be it in vivo or in silico.
