Optimizing Millions of Hyperparameters by Implicit Differentiation
  We propose an algorithm for inexpensive gradient-based hyperparameter
optimization that combines the implicit function theorem (IFT) with efficient
inverse Hessian approximations. We present results about the relationship
between the IFT and differentiating through optimization, motivating our
algorithm. We use the proposed approach to train modern network architectures
with millions of weights and millions of hyper-parameters. For example, we
learn a data-augmentation network - where every weight is a hyperparameter
tuned for validation performance - outputting augmented training examples.
Jointly tuning weights and hyperparameters with our approach is only a few
times more costly in memory and compute than standard training.
