A Distribution-Dependent Analysis of Meta-Learning
  A key problem in the theory of meta-learning is to understand how the task
distributions influence transfer risk, the expected error of a meta-learner on
a new task drawn from the unknown task distribution. In this paper, focusing on
fixed design linear regression with Gaussian noise and a Gaussian task (or
parameter) distribution, we give distribution-dependent lower bounds on the
transfer risk of any algorithm, while we also show that a novel, weighted
version of the so-called biased regularized regression method is able to match
these lower bounds up to a fixed constant factor. Notably, the weighting is
derived from the covariance of the Gaussian task distribution. Altogether, our
results provide a precise characterization of the difficulty of meta-learning
in this Gaussian setting. While this problem setting may appear simple, we show
that it is rich enough to unify the "parameter sharing" and "representation
learning" streams of meta-learning; in particular, representation learning is
obtained as the special case when the covariance matrix of the task
distribution is unknown. For this case we propose to adopt the EM method, which
is shown to enjoy efficient updates in our case. The paper is completed by an
empirical study of EM. In particular, our experimental results show that the EM
algorithm can attain the lower bound as the number of tasks grows, while the
algorithm is also successful in competing with its alternatives when used in a
representation learning context.
