Self-Supervised Learning with Kernel Dependence Maximization
  We approach self-supervised learning of image representations from a
statistical dependence perspective, proposing Self-Supervised Learning with the
Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes
dependence between representations of transformations of an image and the image
identity, while minimizing the kernelized variance of those representations.
This framework yields a new understanding of InfoNCE, a variational lower bound
on the mutual information (MI) between different transformations. While the MI
itself is known to have pathologies which can result in learning meaningless
representations, its bound is much better behaved: we show that it implicitly
approximates SSL-HSIC (with a slightly different regularizer). Our approach
also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC
similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly
optimize statistical dependence in time linear in the batch size, without
restrictive data assumptions or indirect mutual information estimators. Trained
with or without a target network, SSL-HSIC matches the current state-of-the-art
for standard linear evaluation on ImageNet, semi-supervised learning and
transfer to other classification and vision tasks such as semantic
segmentation, depth estimation and object recognition. Code is available at
https://github.com/deepmind/ssl_hsic .
