Meta-Learning with Adaptive Hyperparameters
  Despite its popularity, several recent works question the effectiveness of
MAML when test tasks are different from training tasks, thus suggesting various
task-conditioned methodology to improve the initialization. Instead of
searching for better task-aware initialization, we focus on a complementary
factor in MAML framework, inner-loop optimization (or fast adaptation).
Consequently, we propose a new weight update rule that greatly enhances the
fast adaptation process. Specifically, we introduce a small meta-network that
can adaptively generate per-step hyperparameters: learning rate and weight
decay coefficients. The experimental results validate that the Adaptive
Learning of hyperparameters for Fast Adaptation (ALFA) is the equally important
ingredient that was often neglected in the recent few-shot learning approaches.
Surprisingly, fast adaptation from random initialization with ALFA can already
outperform MAML.
