Exploring Covariate and Concept Shift for Detection and Calibration of
  Out-of-Distribution Data
  Moving beyond testing on in-distribution data works on Out-of-Distribution
(OOD) detection have recently increased in popularity. A recent attempt to
categorize OOD data introduces the concept of near and far OOD detection.
Specifically, prior works define characteristics of OOD data in terms of
detection difficulty. We propose to characterize the spectrum of OOD data using
two types of distribution shifts: covariate shift and concept shift, where
covariate shift corresponds to change in style, e.g., noise, and concept shift
indicates a change in semantics. This characterization reveals that sensitivity
to each type of shift is important to the detection and confidence calibration
of OOD data. Consequently, we investigate score functions that capture
sensitivity to each type of dataset shift and methods that improve them. To
this end, we theoretically derive two score functions for OOD detection, the
covariate shift score and concept shift score, based on the decomposition of
KL-divergence for both scores, and propose a geometrically-inspired method
(Geometric ODIN) to improve OOD detection under both shifts with only
in-distribution data. Additionally, the proposed method naturally leads to an
expressive post-hoc calibration function which yields state-of-the-art
calibration performance on both in-distribution and out-of-distribution data.
We are the first to propose a method that works well across both OOD detection
and calibration and under different types of shifts. View project page at
https://sites.google.com/view/geometric-decomposition.
