VICReg: Variance-Invariance-Covariance Regularization for
  Self-Supervised Learning
  Recent self-supervised methods for image representation learning are based on
maximizing the agreement between embedding vectors from different views of the
same image. A trivial solution is obtained when the encoder outputs constant
vectors. This collapse problem is often avoided through implicit biases in the
learning architecture, that often lack a clear justification or interpretation.
In this paper, we introduce VICReg (Variance-Invariance-Covariance
Regularization), a method that explicitly avoids the collapse problem with a
simple regularization term on the variance of the embeddings along each
dimension individually. VICReg combines the variance term with a decorrelation
mechanism based on redundancy reduction and covariance regularization, and
achieves results on par with the state of the art on several downstream tasks.
In addition, we show that incorporating our new variance term into other
methods helps stabilize the training and leads to performance improvements.
