Representation Learning Beyond Linear Prediction Functions
  Recent papers on the theory of representation learning has shown the
importance of a quantity called diversity when generalizing from a set of
source tasks to a target task. Most of these papers assume that the function
mapping shared representations to predictions is linear, for both source and
target tasks. In practice, researchers in deep learning use different numbers
of extra layers following the pretrained model based on the difficulty of the
new task. This motivates us to ask whether diversity can be achieved when
source tasks and the target task use different prediction function spaces
beyond linear functions. We show that diversity holds even if the target task
uses a neural network with multiple layers, as long as source tasks use linear
functions. If source tasks use nonlinear prediction functions, we provide a
negative result by showing that depth-1 neural networks with ReLu activation
function need exponentially many source tasks to achieve diversity. For a
general function class, we find that eluder dimension gives a lower bound on
the number of tasks required for diversity. Our theoretical results imply that
simpler tasks generalize better. Though our theoretical results are shown for
the global minimizer of empirical risks, their qualitative predictions still
hold true for gradient-based optimization algorithms as verified by our
simulations on deep neural networks.
