ELM: Embedding and Logit Margins for Long-Tail Learning
  Long-tail learning is the problem of learning under skewed label
distributions, which pose a challenge for standard learners. Several recent
approaches for the problem have proposed enforcing a suitable margin in logit
space. Such techniques are intuitive analogues of the guiding principle behind
SVMs, and are equally applicable to linear models and neural models. However,
when applied to neural models, such techniques do not explicitly control the
geometry of the learned embeddings. This can be potentially sub-optimal, since
embeddings for tail classes may be diffuse, resulting in poor generalization
for these classes. We present Embedding and Logit Margins (ELM), a unified
approach to enforce margins in logit space, and regularize the distribution of
embeddings. This connects losses for long-tail learning to proposals in the
literature on metric embedding, and contrastive learning. We theoretically show
that minimising the proposed ELM objective helps reduce the generalisation gap.
The ELM method is shown to perform well empirically, and results in tighter
tail class embeddings.
