Hyperspherically Regularized Networks for Self-Supervision
  Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised
learning avoiding the contrastive paradigm and subsequently removing the
computational burden of negative sampling associated with such methods.
However, we empirically find that the image representations produced under the
BYOL's self-distillation paradigm are poorly distributed in representation
space compared to contrastive methods. This work empirically demonstrates that
feature diversity enforced by contrastive losses is beneficial to image
representation uniformity when employed in BYOL, and as such, provides greater
inter-class representation separability. Additionally, we explore and advocate
the use of regularization methods, specifically the layer-wise minimization of
hyperspherical energy (i.e. maximization of entropy) of network weights to
encourage representation uniformity. We show that directly optimizing a measure
of uniformity alongside the standard loss, or regularizing the networks of the
BYOL architecture to minimize the hyperspherical energy of neurons can produce
more uniformly distributed and therefore better performing representations for
downstream tasks.
