Multi-Task Meta-Learning Modification with Stochastic Approximation
  Meta-learning methods aim to build learning algorithms capable of quickly
adapting to new tasks in low-data regime. One of the main benchmarks of such an
algorithms is a few-shot learning problem. In this paper we investigate the
modification of standard meta-learning pipeline that takes a multi-task
approach during training. The proposed method simultaneously utilizes
information from several meta-training tasks in a common loss function. The
impact of each of these tasks in the loss function is controlled by the
corresponding weight. Proper optimization of these weights can have a big
influence on training of the entire model and might improve the quality on test
time tasks. In this work we propose and investigate the use of methods from the
family of simultaneous perturbation stochastic approximation (SPSA) approaches
for meta-train tasks weights optimization. We have also compared the proposed
algorithms with gradient-based methods and found that stochastic approximation
demonstrates the largest quality boost in test time. Proposed multi-task
modification can be applied to almost all methods that use meta-learning
pipeline. In this paper we study applications of this modification on
Prototypical Networks and Model-Agnostic Meta-Learning algorithms on CIFAR-FS,
FC100, tieredImageNet and miniImageNet few-shot learning benchmarks. During
these experiments, multi-task modification has demonstrated improvement over
original methods. The proposed SPSA-Tracking algorithm shows the largest
accuracy boost that is competitive against the state-of-the-art meta-learning
methods. Our code is available online.
