Tutorial on amortized optimization for learning to optimize over
  continuous domains
  Optimization is a ubiquitous modeling tool and is often deployed in settings
which repeatedly solve similar instances of the same problem. Amortized
optimization methods use learning to predict the solutions to problems in these
settings. This leverages the shared structure between similar problem
instances. In this tutorial, we will discuss the key design choices behind
amortized optimization, roughly categorizing 1) models into fully-amortized and
semi-amortized approaches, and 2) learning methods into regression-based and
objective-based. We then view existing applications through these foundations
to draw connections between them, including for manifold optimization,
variational inference, sparse coding, meta-learning, control, reinforcement
learning, convex optimization, and deep equilibrium networks. This framing
enables us easily see, for example, that the amortized inference in variational
autoencoders is conceptually identical to value gradients in control and
reinforcement learning as they both use fully-amortized models with an
objective-based loss. The source code for this tutorial is available at
https://www.github.com/facebookresearch/amortized-optimization-tutorial
