On the Iteration Complexity of Hypergradient Computation
  We study a general class of bilevel problems, consisting in the minimization
of an upper-level objective which depends on the solution to a parametric
fixed-point equation. Important instances arising in machine learning include
hyperparameter optimization, meta-learning, and certain graph and recurrent
neural networks. Typically the gradient of the upper-level objective
(hypergradient) is hard or even impossible to compute exactly, which has raised
the interest in approximation methods. We investigate some popular approaches
to compute the hypergradient, based on reverse mode iterative differentiation
and approximate implicit differentiation. Under the hypothesis that the fixed
point equation is defined by a contraction mapping, we present a unified
analysis which allows for the first time to quantitatively compare these
methods, providing explicit bounds for their iteration complexity. This
analysis suggests a hierarchy in terms of computational efficiency among the
above methods, with approximate implicit differentiation based on conjugate
gradient performing best. We present an extensive experimental comparison among
the methods which confirm the theoretical findings.
