Functional Regularization for Representation Learning: A Unified
  Theoretical Perspective
  Unsupervised and self-supervised learning approaches have become a crucial
tool to learn representations for downstream prediction tasks. While these
approaches are widely used in practice and achieve impressive empirical gains,
their theoretical understanding largely lags behind. Towards bridging this gap,
we present a unifying perspective where several such approaches can be viewed
as imposing a regularization on the representation via a learnable function
using unlabeled data. We propose a discriminative theoretical framework for
analyzing the sample complexity of these approaches, which generalizes the
framework of (Balcan and Blum, 2010) to allow learnable regularization
functions. Our sample complexity bounds show that, with carefully chosen
hypothesis classes to exploit the structure in the data, these learnable
regularization functions can prune the hypothesis space, and help reduce the
amount of labeled data needed. We then provide two concrete examples of
functional regularization, one using auto-encoders and the other using masked
self-supervision, and apply our framework to quantify the reduction in the
sample complexity bound of labeled data. We also provide complementary
empirical results to support our analysis.
