Double Double Descent: On Generalization Errors in Transfer Learning
  between Linear Regression Tasks
  We study the transfer learning process between two linear regression
problems. An important and timely special case is when the regressors are
overparameterized and perfectly interpolate their training data. We examine a
parameter transfer mechanism whereby a subset of the parameters of the target
task solution are constrained to the values learned for a related source task.
We analytically characterize the generalization error of the target task in
terms of the salient factors in the transfer learning architecture, i.e., the
number of examples available, the number of (free) parameters in each of the
tasks, the number of parameters transferred from the source to target task, and
the relation between the two tasks. Our non-asymptotic analysis shows that the
generalization error of the target task follows a two-dimensional double
descent trend (with respect to the number of free parameters in each of the
tasks) that is controlled by the transfer learning factors. Our analysis points
to specific cases where the transfer of parameters is beneficial as a
substitute for extra overparameterization (i.e., additional free parameters in
the target task). Specifically, we show that the usefulness of a transfer
learning setting is fragile and depends on a delicate interplay among the set
of transferred parameters, the relation between the tasks, and the true
solution. We also demonstrate that overparameterized transfer learning is not
necessarily more beneficial when the source task is closer or identical to the
target task.
