Bootstrapped Meta-Learning
  Meta-learning empowers artificial intelligence to increase its efficiency by
learning how to learn. Unlocking this potential involves overcoming a
challenging meta-optimisation problem. We propose an algorithm that tackles
this problem by letting the meta-learner teach itself. The algorithm first
bootstraps a target from the meta-learner, then optimises the meta-learner by
minimising the distance to that target under a chosen (pseudo-)metric. Focusing
on meta-learning with gradients, we establish conditions that guarantee
performance improvements and show that the metric can control
meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the
effective meta-learning horizon without requiring backpropagation through all
updates. We achieve a new state-of-the art for model-free agents on the Atari
ALE benchmark and demonstrate that it yields both performance and efficiency
gains in multi-task meta-learning. Finally, we explore how bootstrapping opens
up new possibilities and find that it can meta-learn efficient exploration in
an epsilon-greedy Q-learning agent, without backpropagating through the update
rule.
