Simple and Effective Balance of Contrastive Losses
  Contrastive losses have long been a key ingredient of deep metric learning
and are now becoming more popular due to the success of self-supervised
learning. Recent research has shown the benefit of decomposing such losses into
two sub-losses which act in a complementary way when learning the
representation network: a positive term and an entropy term. Although the
overall loss is thus defined as a combination of two terms, the balance of
these two terms is often hidden behind implementation details and is largely
ignored and sub-optimal in practice. In this work, we approach the balance of
contrastive losses as a hyper-parameter optimization problem, and propose a
coordinate descent-based search method that efficiently find the
hyper-parameters that optimize evaluation performance. In the process, we
extend existing balance analyses to the contrastive margin loss, include batch
size in the balance, and explain how to aggregate loss elements from the batch
to maintain near-optimal performance over a larger range of batch sizes.
Extensive experiments with benchmarks from deep metric learning and
self-supervised learning show that optimal hyper-parameters are found faster
with our method than with other common search methods.
