Backpropagation in the Simply Typed Lambda-calculus with Linear Negation
  Backpropagation is a classic automatic differentiation algorithm computing
the gradient of functions specified by a certain class of simple, first-order
programs, called computational graphs. It is a fundamental tool in several
fields, most notably machine learning, where it is the key for efficiently
training (deep) neural networks. Recent years have witnessed the quick growth
of a research field called differentiable programming, the aim of which is to
express computational graphs more synthetically and modularly by resorting to
actual programming languages endowed with control flow operators and
higher-order combinators, such as map and fold. In this paper, we extend the
backpropagation algorithm to a paradigmatic example of such a programming
language: we define a compositional program transformation from the
simply-typed lambda-calculus to itself augmented with a notion of linear
negation, and prove that this computes the gradient of the source program with
the same efficiency as first-order backpropagation. The transformation is
completely effect-free and thus provides a purely logical understanding of the
dynamics of backpropagation.
