Adapted Deep Embeddings: A Synthesis of Methods for $k$-Shot Inductive
  Transfer Learning
  The focus in machine learning has branched beyond training classifiers on a
single task to investigating how previously acquired knowledge in a source
domain can be leveraged to facilitate learning in a related target domain,
known as inductive transfer learning. Three active lines of research have
independently explored transfer learning using neural networks. In weight
transfer, a model trained on the source domain is used as an initialization
point for a network to be trained on the target domain. In deep metric
learning, the source domain is used to construct an embedding that captures
class structure in both the source and target domains. In few-shot learning,
the focus is on generalizing well in the target domain based on a limited
number of labeled examples. We compare state-of-the-art methods from these
three paradigms and also explore hybrid adapted-embedding methods that use
limited target-domain data to fine tune embeddings constructed from
source-domain data. We conduct a systematic comparison of methods in a variety
of domains, varying the number of labeled instances available in the target
domain ($k$), as well as the number of target-domain classes. We reach three
principal conclusions: (1) Deep embeddings are far superior, compared to weight
transfer, as a starting point for inter-domain transfer or model re-use (2) Our
hybrid methods robustly outperform every few-shot learning and every deep
metric learning method previously proposed, with a mean error reduction of 34%
over state-of-the-art. (3) Among loss functions for discovering embeddings, the
histogram loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results
will motivate a unification of research in weight transfer, deep metric
learning, and few-shot learning.
