Robust Contrastive Learning Using Negative Samples with Diminished
  Semantics
  Unsupervised learning has recently made exceptional progress because of the
development of more effective contrastive learning methods. However, CNNs are
prone to depend on low-level features that humans deem non-semantic. This
dependency has been conjectured to induce a lack of robustness to image
perturbations or domain shift. In this paper, we show that by generating
carefully designed negative samples, contrastive learning can learn more robust
representations with less dependence on such features. Contrastive learning
utilizes positive pairs that preserve semantic information while perturbing
superficial features in the training images. Similarly, we propose to generate
negative samples in a reversed way, where only the superfluous instead of the
semantic features are preserved. We develop two methods, texture-based and
patch-based augmentations, to generate negative samples. These samples achieve
better generalization, especially under out-of-domain settings. We also analyze
our method and the generated texture-based samples, showing that texture
features are indispensable in classifying particular ImageNet classes and
especially finer classes. We also show that model bias favors texture and shape
features differently under different test settings. Our code, trained models,
and ImageNet-Texture dataset can be found at
https://github.com/SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives.
