Object-aware Contrastive Learning for Debiased Scene Representation
  Contrastive self-supervised learning has shown impressive results in learning
visual representations from unlabeled images by enforcing invariance against
different data augmentations. However, the learned representations are often
contextually biased to the spurious scene correlations of different objects or
object and background, which may harm their generalization on the downstream
tasks. To tackle the issue, we develop a novel object-aware contrastive
learning framework that first (a) localizes objects in a self-supervised manner
and then (b) debias scene correlations via appropriate data augmentations
considering the inferred object locations. For (a), we propose the contrastive
class activation map (ContraCAM), which finds the most discriminative regions
(e.g., objects) in the image compared to the other images using the
contrastively trained models. We further improve the ContraCAM to detect
multiple objects and entire shapes via an iterative refinement procedure. For
(b), we introduce two data augmentations based on ContraCAM, object-aware
random crop and background mixup, which reduce contextual and background biases
during contrastive self-supervised learning, respectively. Our experiments
demonstrate the effectiveness of our representation learning framework,
particularly when trained under multi-object images or evaluated under the
background (and distribution) shifted images.
