Bilevel Programming for Hyperparameter Optimization and Meta-Learning
  We introduce a framework based on bilevel programming that unifies
gradient-based hyperparameter optimization and meta-learning. We show that an
approximate version of the bilevel problem can be solved by taking into
explicit account the optimization dynamics for the inner objective. Depending
on the specific setting, the outer variables take either the meaning of
hyperparameters in a supervised learning problem or parameters of a
meta-learner. We provide sufficient conditions under which solutions of the
approximate problem converge to those of the exact problem. We instantiate our
approach for meta-learning in the case of deep learning where representation
layers are treated as hyperparameters shared across a set of training episodes.
In experiments, we confirm our theoretical findings, present encouraging
results for few-shot learning and contrast the bilevel approach against
classical approaches for learning-to-learn.
