Natural Attribute-based Shift Detection
  Despite the impressive performance of deep networks in vision, language, and
healthcare, unpredictable behaviors on samples from the distribution different
than the training distribution cause severe problems in deployment. For better
reliability of neural-network-based classifiers, we define a new task, natural
attribute-based shift (NAS) detection, to detect the samples shifted from the
training distribution by some natural attribute such as age of subjects or
brightness of images. Using the natural attributes present in existing
datasets, we introduce benchmark datasets in vision, language, and medical for
NAS detection. Further, we conduct an extensive evaluation of prior
representative out-of-distribution (OOD) detection methods on NAS datasets and
observe an inconsistency in their performance. To understand this, we provide
an analysis on the relationship between the location of NAS samples in the
feature space and the performance of distance- and confidence-based OOD
detection methods. Based on the analysis, we split NAS samples into three
categories and further suggest a simple modification to the training objective
to obtain an improved OOD detection method that is capable of detecting samples
from all NAS categories.
