HIDRA: Head Initialization across Dynamic targets for Robust
  Architectures
  The performance of gradient-based optimization strategies depends heavily on
the initial weights of the parametric model. Recent works show that there exist
weight initializations from which optimization procedures can find the
task-specific parameters faster than from uniformly random initializations and
that such a weight initialization can be learned by optimizing a specific model
architecture across similar tasks via MAML (Model-Agnostic Meta-Learning).
Current methods are limited to populations of classification tasks that share
the same number of classes due to the static model architectures used during
meta-learning. In this paper, we present HIDRA, a meta-learning approach that
enables training and evaluating across tasks with any number of target
variables. We show that Model-Agnostic Meta-Learning trains a distribution for
all the neurons in the output layer and a specific weight initialization for
the ones in the hidden layers. HIDRA explores this by learning one master
neuron, which is used to initialize any number of output neurons for a new
task. Extensive experiments on the Miniimagenet and Omniglot data sets
demonstrate that HIDRA improves over standard approaches while generalizing to
tasks with any number of target variables. Moreover, our approach is shown to
robustify low-capacity models in learning across complex tasks with a high
number of classes for which regular MAML fails to learn any feasible
initialization.
