Toward Understanding the Feature Learning Process of Self-supervised
  Contrastive Learning
  How can neural networks trained by contrastive learning extract features from
the unlabeled data? Why does contrastive learning usually need much stronger
data augmentations than supervised learning to ensure good representations?
These questions involve both the optimization and statistical aspects of deep
learning, but can hardly be answered by analyzing supervised learning, where
the target functions are the highest pursuit. Indeed, in self-supervised
learning, it is inevitable to relate to the optimization/generalization of
neural networks to how they can encode the latent structures in the data, which
we refer to as the feature learning process.
  In this work, we formally study how contrastive learning learns the feature
representations for neural networks by analyzing its feature learning process.
We consider the case where our data are comprised of two types of features: the
more semantically aligned sparse features which we want to learn from, and the
other dense features we want to avoid. Theoretically, we prove that contrastive
learning using $\mathbf{ReLU}$ networks provably learns the desired sparse
features if proper augmentations are adopted. We present an underlying
principle called $\textbf{feature decoupling}$ to explain the effects of
augmentations, where we theoretically characterize how augmentations can reduce
the correlations of dense features between positive samples while keeping the
correlations of sparse features intact, thereby forcing the neural networks to
learn from the self-supervision of sparse features. Empirically, we verified
that the feature decoupling principle matches the underlying mechanism of
contrastive learning in practice.
