Meta-Learning and Universality: Deep Representations and Gradient
  Descent can Approximate any Learning Algorithm
  Learning to learn is a powerful paradigm for enabling models to learn from
data more effectively and efficiently. A popular approach to meta-learning is
to train a recurrent model to read in a training dataset as input and output
the parameters of a learned model, or output predictions for new test inputs.
Alternatively, a more recent approach to meta-learning aims to acquire deep
representations that can be effectively fine-tuned, via standard gradient
descent, to new tasks. In this paper, we consider the meta-learning problem
from the perspective of universality, formalizing the notion of learning
algorithm approximation and comparing the expressive power of the
aforementioned recurrent models to the more recent approaches that embed
gradient descent into the meta-learner. In particular, we seek to answer the
following question: does deep representation combined with standard gradient
descent have sufficient capacity to approximate any learning algorithm? We find
that this is indeed true, and further find, in our experiments, that
gradient-based meta-learning consistently leads to learning strategies that
generalize more widely compared to those represented by recurrent models.
