Categorical Foundations of Gradient-Based Learning
  We propose a categorical semantics of gradient-based machine learning
algorithms in terms of lenses, parametrised maps, and reverse derivative
categories. This foundation provides a powerful explanatory and unifying
framework: it encompasses a variety of gradient descent algorithms such as
ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions
such as as MSE and Softmax cross-entropy, shedding new light on their
similarities and differences. Our approach to gradient-based learning has
examples generalising beyond the familiar continuous domains (modelled in
categories of smooth maps) and can be realized in the discrete setting of
boolean circuits. Finally, we demonstrate the practical significance of our
framework with an implementation in Python.
