Training End-to-End Analog Neural Networks with Equilibrium Propagation
  We introduce a principled method to train end-to-end analog neural networks
by stochastic gradient descent. In these analog neural networks, the weights to
be adjusted are implemented by the conductances of programmable resistive
devices such as memristors [Chua, 1971], and the nonlinear transfer functions
(or `activation functions') are implemented by nonlinear components such as
diodes. We show mathematically that a class of analog neural networks (called
nonlinear resistive networks) are energy-based models: they possess an energy
function as a consequence of Kirchhoff's laws governing electrical circuits.
This property enables us to train them using the Equilibrium Propagation
framework [Scellier and Bengio, 2017]. Our update rule for each conductance,
which is local and relies solely on the voltage drop across the corresponding
resistor, is shown to compute the gradient of the loss function. Our numerical
simulations, which use the SPICE-based Spectre simulation framework to simulate
the dynamics of electrical circuits, demonstrate training on the MNIST
classification task, performing comparably or better than equivalent-size
software-based neural networks. Our work can guide the development of a new
generation of ultra-fast, compact and low-power neural networks supporting
on-chip learning.
