Investigating Power laws in Deep Representation Learning
  Representation learning that leverages large-scale labelled datasets, is
central to recent progress in machine learning. Access to task relevant labels
at scale is often scarce or expensive, motivating the need to learn from
unlabelled datasets with self-supervised learning (SSL). Such large unlabelled
datasets (with data augmentations) often provide a good coverage of the
underlying input distribution. However evaluating the representations learned
by SSL algorithms still requires task-specific labelled samples in the training
pipeline. Additionally, the generalization of task-specific encoding is often
sensitive to potential distribution shift. Inspired by recent advances in
theoretical machine learning and vision neuroscience, we observe that the
eigenspectrum of the empirical feature covariance matrix often follows a power
law. For visual representations, we estimate the coefficient of the power law,
$\alpha$, across three key attributes which influence representation learning:
learning objective (supervised, SimCLR, Barlow Twins and BYOL), network
architecture (VGG, ResNet and Vision Transformer), and tasks (object and scene
recognition). We observe that under mild conditions, proximity of $\alpha$ to
1, is strongly correlated to the downstream generalization performance.
Furthermore, $\alpha \approx 1$ is a strong indicator of robustness to label
noise during fine-tuning. Notably, $\alpha$ is computable from the
representations without knowledge of any labels, thereby offering a framework
to evaluate the quality of representations in unlabelled datasets.
