Entropic Out-of-Distribution Detection: Seamless Detection of Unknown
  Examples
  In this paper, we argue that the unsatisfactory out-of-distribution (OOD)
detection performance of neural networks is mainly due to the SoftMax loss
anisotropy and propensity to produce low entropy probability distributions in
disagreement with the principle of maximum entropy. Current out-of-distribution
(OOD) detection approaches usually do not directly fix the SoftMax loss
drawbacks, but rather build techniques to circumvent it. Unfortunately, those
methods usually produce undesired side effects (e.g., classification accuracy
drop, additional hyperparameters, slower inferences, and collecting extra
data). In the opposite direction, we propose replacing SoftMax loss with a
novel loss function that does not suffer from the mentioned weaknesses. The
proposed IsoMax loss is isotropic (exclusively distance-based) and provides
high entropy posterior probability distributions. Replacing the SoftMax loss by
IsoMax loss requires no model or training changes. Additionally, the models
trained with IsoMax loss produce as fast and energy-efficient inferences as
those trained using SoftMax loss. Moreover, no classification accuracy drop is
observed. The proposed method does not rely on outlier/background data,
hyperparameter tuning, temperature calibration, feature extraction, metric
learning, adversarial training, ensemble procedures, or generative models. Our
experiments showed that IsoMax loss works as a seamless SoftMax loss drop-in
replacement that significantly improves neural networks' OOD detection
performance. Hence, it may be used as a baseline OOD detection approach to be
combined with current or future OOD detection techniques to achieve even higher
results.
