Automatic Open-World Reliability Assessment
  Image classification in the open-world must handle out-of-distribution (OOD)
images. Systems should ideally reject OOD images, or they will map atop of
known classes and reduce reliability. Using open-set classifiers that can
reject OOD inputs can help. However, optimal accuracy of open-set classifiers
depend on the frequency of OOD data. Thus, for either standard or open-set
classifiers, it is important to be able to determine when the world changes and
increasing OOD inputs will result in reduced system reliability. However,
during operations, we cannot directly assess accuracy as there are no labels.
Thus, the reliability assessment of these classifiers must be done by human
operators, made more complex because networks are not 100% accurate, so some
failures are to be expected. To automate this process, herein, we formalize the
open-world recognition reliability problem and propose multiple automatic
reliability assessment policies to address this new problem using only the
distribution of reported scores/probability data. The distributional algorithms
can be applied to both classic classifiers with SoftMax as well as the
open-world Extreme Value Machine (EVM) to provide automated reliability
assessment. We show that all of the new algorithms significantly outperform
detection using the mean of SoftMax.
