BOIL: Towards Representation Change for Few-shot Learning
  Model Agnostic Meta-Learning (MAML) is one of the most representative of
gradient-based meta-learning algorithms. MAML learns new tasks with a few data
samples using inner updates from a meta-initialization point and learns the
meta-initialization parameters with outer updates. It has recently been
hypothesized that representation reuse, which makes little change in efficient
representations, is the dominant factor in the performance of the
meta-initialized model through MAML in contrast to representation change, which
causes a significant change in representations. In this study, we investigate
the necessity of representation change for the ultimate goal of few-shot
learning, which is solving domain-agnostic tasks. To this aim, we propose a
novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop),
which updates only the body (extractor) of the model and freezes the head
(classifier) during inner loop updates. BOIL leverages representation change
rather than representation reuse. This is because feature vectors
(representations) have to move quickly to their corresponding frozen head
vectors. We visualize this property using cosine similarity, CKA, and empirical
results without the head. BOIL empirically shows significant performance
improvement over MAML, particularly on cross-domain tasks. The results imply
that representation change in gradient-based meta-learning approaches is a
critical component.
