Hierarchical Variational Memory for Few-shot Learning Across Domains
  Neural memory enables fast adaptation to new tasks with just a few training
samples. Existing memory models store features only from the single last layer,
which does not generalize well in presence of a domain shift between training
and test distributions. Rather than relying on a flat memory, we propose a
hierarchical alternative that stores features at different semantic levels. We
introduce a hierarchical prototype model, where each level of the prototype
fetches corresponding information from the hierarchical memory. The model is
endowed with the ability to flexibly rely on features at different semantic
levels if the domain shift circumstances so demand. We meta-learn the model by
a newly derived hierarchical variational inference framework, where
hierarchical memory and prototypes are jointly optimized. To explore and
exploit the importance of different semantic levels, we further propose to
learn the weights associated with the prototype at each level in a data-driven
way, which enables the model to adaptively choose the most generalizable
features. We conduct thorough ablation studies to demonstrate the effectiveness
of each component in our model. The new state-of-the-art performance on
cross-domain and competitive performance on traditional few-shot classification
further substantiates the benefit of hierarchical variational memory.
