Modeling and Optimization Trade-off in Meta-learning
  By searching for shared inductive biases across tasks, meta-learning promises
to accelerate learning on novel tasks, but with the cost of solving a complex
bilevel optimization problem. We introduce and rigorously define the trade-off
between accurate modeling and optimization ease in meta-learning. At one end,
classic meta-learning algorithms account for the structure of meta-learning but
solve a complex optimization problem, while at the other end domain randomized
search (otherwise known as joint training) ignores the structure of
meta-learning and solves a single level optimization problem. Taking MAML as
the representative meta-learning algorithm, we theoretically characterize the
trade-off for general non-convex risk functions as well as linear regression,
for which we are able to provide explicit bounds on the errors associated with
modeling and optimization. We also empirically study this trade-off for
meta-reinforcement learning benchmarks.
