Conditional Contrastive Learning: Removing Undesirable Information in
  Self-Supervised Representations
  Self-supervised learning is a form of unsupervised learning that leverages
rich information in data to learn representations. However, data sometimes
contains certain information that may be undesirable for downstream tasks. For
instance, gender information may lead to biased decisions on many
gender-irrelevant tasks. In this paper, we develop conditional contrastive
learning to remove undesirable information in self-supervised representations.
To remove the effect of the undesirable variable, our proposed approach
conditions on the undesirable variable (i.e., by fixing the variations of it)
during the contrastive learning process. In particular, inspired by the
contrastive objective InfoNCE, we introduce Conditional InfoNCE (C-InfoNCE),
and its computationally efficient variant, Weak-Conditional InfoNCE
(WeaC-InfoNCE), for conditional contrastive learning. We demonstrate
empirically that our methods can successfully learn self-supervised
representations for downstream tasks while removing a great level of
information related to the undesirable variables. We study three scenarios,
each with a different type of undesirable variables: task-irrelevant
meta-information for self-supervised speech representation learning, sensitive
attributes for fair representation learning, and domain specification for
multi-domain visual representation learning.
