Towards Sample-efficient Overparameterized Meta-learning
  An overarching goal in machine learning is to build a generalizable model
with few samples. To this end, overparameterization has been the subject of
immense interest to explain the generalization ability of deep nets even when
the size of the dataset is smaller than that of the model. While the prior
literature focuses on the classical supervised setting, this paper aims to
demystify overparameterization for meta-learning. Here we have a sequence of
linear-regression tasks and we ask: (1) Given earlier tasks, what is the
optimal linear representation of features for a new downstream task? and (2)
How many samples do we need to build this representation? This work shows that
surprisingly, overparameterization arises as a natural answer to these
fundamental meta-learning questions. Specifically, for (1), we first show that
learning the optimal representation coincides with the problem of designing a
task-aware regularization to promote inductive bias. We leverage this inductive
bias to explain how the downstream task actually benefits from
overparameterization, in contrast to prior works on few-shot learning. For (2),
we develop a theory to explain how feature covariance can implicitly help
reduce the sample complexity well below the degrees of freedom and lead to
small estimation error. We then integrate these findings to obtain an overall
performance guarantee for our meta-learning algorithm. Numerical experiments on
real and synthetic data verify our insights on overparameterized meta-learning.
