Distinction Maximization Loss: Efficiently Improving Classification Accuracy, Uncertainty Estimation, and Out-of-Distribution Detection Simply Replacing the Loss and Calibrating
Building robust deterministic neural networks remains a challenge. On the one
hand, some approaches improve out-of-distribution detection at the cost of
reducing classification accuracy in some situations. On the other hand, some
methods simultaneously increase classification accuracy, uncertainty
estimation, and out-of-distribution detection at the expense of reducing the
inference efficiency and requiring training the same model many times to tune
hyperparameters. In this paper, we propose training deterministic neural
networks using our DisMax loss, which works as a drop-in replacement for the
usual SoftMax loss (i.e., the combination of the linear output layer, the
SoftMax activation, and the cross-entropy loss). Starting from the IsoMax+
loss, we create each logit based on the distances to all prototypes rather than
just the one associated with the correct class. We also introduce a mechanism
to combine images to construct what we call fractional probability
regularization. Moreover, we present a fast way to calibrate the network after
training. Finally, we propose a composite score to perform out-of-distribution
detection. Our experiments show that DisMax usually outperforms current
approaches simultaneously in classification accuracy, uncertainty estimation,
and out-of-distribution detection while maintaining deterministic neural
network inference efficiency and avoiding training the same model repetitively
for hyperparameter tuning. The code to reproduce the results is available at
https://github.com/dlmacedo/distinction-maximization-loss.