Generalized Optimization: A First Step Towards Category Theoretic
  Learning Theory
  The Cartesian reverse derivative is a categorical generalization of
reverse-mode automatic differentiation. We use this operator to generalize
several optimization algorithms, including a straightforward generalization of
gradient descent and a novel generalization of Newton's method. We then explore
which properties of these algorithms are preserved in this generalized setting.
First, we show that the transformation invariances of these algorithms are
preserved: while generalized Newton's method is invariant to all invertible
linear transformations, generalized gradient descent is invariant only to
orthogonal linear transformations. Next, we show that we can express the change
in loss of generalized gradient descent with an inner product-like expression,
thereby generalizing the non-increasing and convergence properties of the
gradient descent optimization flow. Finally, we include several numerical
experiments to illustrate the ideas in the paper and demonstrate how we can use
them to optimize polynomial functions over an ordered ring.
