On the Memorization Properties of Contrastive Learning
  Memorization studies of deep neural networks (DNNs) help to understand what
patterns and how do DNNs learn, and motivate improvements to DNN training
approaches. In this work, we investigate the memorization properties of SimCLR,
a widely used contrastive self-supervised learning approach, and compare them
to the memorization of supervised learning and random labels training. We find
that both training objects and augmentations may have different complexity in
the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar
to random labels training in terms of the distribution of training objects
complexity.
