Run Away From your Teacher: Understanding BYOL by a Novel
  Self-Supervised Approach
  Recently, a newly proposed self-supervised framework Bootstrap Your Own
Latent (BYOL) seriously challenges the necessity of negative samples in
contrastive learning frameworks. BYOL works like a charm despite the fact that
it discards the negative samples completely and there is no measure to prevent
collapse in its training objective. In this paper, we suggest understanding
BYOL from the view of our proposed interpretable self-supervised learning
framework, Run Away From your Teacher (RAFT). RAFT optimizes two objectives at
the same time: (i) aligning two views of the same data to similar
representations and (ii) running away from the model's Mean Teacher (MT, the
exponential moving average of the history models) instead of BYOL's running
towards it. The second term of RAFT explicitly prevents the representation
collapse and thus makes RAFT a more conceptually reliable framework. We provide
basic benchmarks of RAFT on CIFAR10 to validate the effectiveness of our
method. Furthermore, we prove that BYOL is equivalent to RAFT under certain
conditions, providing solid reasoning for BYOL's counter-intuitive success.
