Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic
  Choices
  The area under the ROC curve (AUROC) has been vigorously applied for
imbalanced classification and moreover combined with deep learning techniques.
However, there is no existing work that provides sound information for peers to
choose appropriate deep AUROC maximization techniques. In this work, we fill
this gap from three aspects. (i) We benchmark a variety of loss functions with
different algorithmic choices for deep AUROC optimization problem. We study the
loss functions in two categories: pairwise loss and composite loss, which
includes a total of 10 loss functions. Interestingly, we find composite loss,
as an innovative loss function class, shows more competitive performance than
pairwise loss from both training convergence and testing generalization
perspectives. Nevertheless, data with more corrupted labels favors a pairwise
symmetric loss. (ii) Moreover, we benchmark and highlight the essential
algorithmic choices such as positive sampling rate, regularization,
normalization/activation, and optimizers. Key findings include: higher positive
sampling rate is likely to be beneficial for deep AUROC maximization; different
datasets favors different weights of regularizations; appropriate normalization
techniques, such as sigmoid and $\ell_2$ score normalization, could improve
model performance. (iii) For optimization aspect, we benchmark SGD-type,
Momentum-type, and Adam-type optimizers for both pairwise and composite loss.
Our findings show that although Adam-type method is more competitive from
training perspective, but it does not outperform others from testing
perspective.
