Multi-step Estimation for Gradient-based Meta-learning
  Gradient-based meta-learning approaches have been successful in few-shot
learning, transfer learning, and a wide range of other domains. Despite its
efficacy and simplicity, the burden of calculating the Hessian matrix with
large memory footprints is the critical challenge in large-scale applications.
To tackle this issue, we propose a simple yet straightforward method to reduce
the cost by reusing the same gradient in a window of inner steps. We describe
the dynamics of the multi-step estimation in the Lagrangian formalism and
discuss how to reduce evaluating second-order derivatives estimating the
dynamics. To validate our method, we experiment on meta-transfer learning and
few-shot learning tasks for multiple settings. The experiment on meta-transfer
emphasizes the applicability of training meta-networks, where other
approximations are limited. For few-shot learning, we evaluate time and memory
complexities compared with popular baselines. We show that our method
significantly reduces training time and memory usage, maintaining competitive
accuracies, or even outperforming in some cases.
