A Modular Theory of Feature Learning
  Learning representations of data, and in particular learning features for a
subsequent prediction task, has been a fruitful area of research delivering
impressive empirical results in recent years. However, relatively little is
understood about what makes a representation `good'. We propose the idea of a
risk gap induced by representation learning for a given prediction context,
which measures the difference in the risk of some learner using the learned
features as compared to the original inputs. We describe a set of sufficient
conditions for unsupervised representation learning to provide a benefit, as
measured by this risk gap. These conditions decompose the problem of when
representation learning works into its constituent parts, which can be
separately evaluated using an unlabeled sample, suitable domain-specific
assumptions about the joint distribution, and analysis of the feature learner
and subsequent supervised learner. We provide two examples of such conditions
in the context of specific properties of the unlabeled distribution, namely
when the data lies close to a low-dimensional manifold and when it forms
clusters. We compare our approach to a recently proposed analysis of
semi-supervised learning.
