Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
  Previous work shows that adversarially robust generalization requires larger
sample complexity, and the same dataset, e.g., CIFAR-10, which enables good
standard accuracy may not suffice to train robust models. Since collecting new
training data could be costly, we focus on better utilizing the given data by
inducing the regions with high sample density in the feature space, which could
lead to locally sufficient samples for robust learning. We first formally show
that the softmax cross-entropy (SCE) loss and its variants convey inappropriate
supervisory signals, which encourage the learned feature points to spread over
the space sparsely in training. This inspires us to propose the Max-Mahalanobis
center (MMC) loss to explicitly induce dense feature regions in order to
benefit robustness. Namely, the MMC loss encourages the model to concentrate on
learning ordered and compact representations, which gather around the preset
optimal centers for different classes. We empirically demonstrate that applying
the MMC loss can significantly improve robustness even under strong adaptive
attacks, while keeping state-of-the-art accuracy on clean inputs with little
extra computation compared to the SCE loss.
