Don't Judge an Object by Its Context: Learning to Overcome Contextual
  Bias
  Existing models often leverage co-occurrences between objects and their
context to improve recognition accuracy. However, strongly relying on context
risks a model's generalizability, especially when typical co-occurrence
patterns are absent. This work focuses on addressing such contextual biases to
improve the robustness of the learnt feature representations. Our goal is to
accurately recognize a category in the absence of its context, without
compromising on performance when it co-occurs with context. Our key idea is to
decorrelate feature representations of a category from its co-occurring
context. We achieve this by learning a feature subspace that explicitly
represents categories occurring in the absence of context along side a joint
feature subspace that represents both categories and context. Our very simple
yet effective method is extensible to two multi-label tasks -- object and
attribute classification. On 4 challenging datasets, we demonstrate the
effectiveness of our method in reducing contextual bias.
