Single Layer Predictive Normalized Maximum Likelihood for
  Out-of-Distribution Detection
  Detecting out-of-distribution (OOD) samples is vital for developing machine
learning based models for critical safety systems. Common approaches for OOD
detection assume access to some OOD samples during training which may not be
available in a real-life scenario. Instead, we utilize the {\em predictive
normalized maximum likelihood} (pNML) learner, in which no assumptions are made
on the tested input. We derive an explicit expression of the pNML and its
generalization error, denoted as the {\em regret}, for a single layer neural
network (NN). We show that this learner generalizes well when (i) the test
vector resides in a subspace spanned by the eigenvectors associated with the
large eigenvalues of the empirical correlation matrix of the training data, or
(ii) the test sample is far from the decision boundary. Furthermore, we
describe how to efficiently apply the derived pNML regret to any pretrained
deep NN, by employing the explicit pNML for the last layer, followed by the
softmax function. Applying the derived regret to deep NN requires neither
additional tunable parameters nor extra data. We extensively evaluate our
approach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and
WideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30
showing a significant improvement of up to 15.6\% over recent leading methods.
