Planckian jitter: enhancing the color quality of self-supervised visual
  representations
  Several recent works on self-supervised learning are trained by mapping
different augmentations of the same image to the same feature representation.
The set of used data augmentations is of crucial importance for the quality of
the learned feature representation. We analyze how the traditionally used color
jitter negatively impacts the quality of the color features in the learned
feature representation. To address this problem, we replace this module with
physics-based color augmentation, called Planckian jitter, which creates
realistic variations in chromaticity, producing a model robust to llumination
changes that can be commonly observed in real life, while maintaining the
ability to discriminate the image content based on color information. We
further improve the performance by introducing a latent space combination of
color-sensitive and non-color-sensitive features. These are found to be
complementary and the combination leads to large absolute performance gains
over the default data augmentation on color classification tasks, including on
Flowers-102 (+15%), Cub200 (+11%), VegFru (+15%), and T1K+ (+12%). Finally, we
present a color sensitivity analysis to document the impact of different
training methods on the model neurons and we show that the performance of the
learned features is robust with respect to illuminant variations.
