Solving Inefficiency of Self-supervised Representation Learning
  Self-supervised learning (especially contrastive learning) has attracted
great interest due to its huge potential in learning discriminative
representations in an unsupervised manner. Despite the acknowledged successes,
existing contrastive learning methods suffer from very low learning efficiency,
e.g., taking about ten times more training epochs than supervised learning for
comparable recognition accuracy. In this paper, we reveal two contradictory
phenomena in contrastive learning that we call under-clustering and
over-clustering problems, which are major obstacles to learning efficiency.
Under-clustering means that the model cannot efficiently learn to discover the
dissimilarity between inter-class samples when the negative sample pairs for
contrastive learning are insufficient to differentiate all the actual object
classes. Over-clustering implies that the model cannot efficiently learn
features from excessive negative sample pairs, forcing the model to
over-cluster samples of the same actual classes into different clusters. To
simultaneously overcome these two problems, we propose a novel self-supervised
learning framework using a truncated triplet loss. Precisely, we employ a
triplet loss tending to maximize the relative distance between the positive
pair and negative pairs to address the under-clustering problem; and we
construct the negative pair by selecting a negative sample deputy from all
negative samples to avoid the over-clustering problem, guaranteed by the
Bernoulli Distribution model. We extensively evaluate our framework in several
large-scale benchmarks (e.g., ImageNet, SYSU-30k, and COCO). The results
demonstrate our model's superiority (e.g., the learning efficiency) over the
latest state-of-the-art methods by a clear margin. Codes available at:
https://github.com/wanggrun/triplet .
