Detecting Underspecification with Local Ensembles
  We present local ensembles, a method for detecting underspecification -- when
many possible predictors are consistent with the training data and model class
-- at test time in a pre-trained model. Our method uses local second-order
information to approximate the variance of predictions across an ensemble of
models from the same class. We compute this approximation by estimating the
norm of the component of a test point's gradient that aligns with the
low-curvature directions of the Hessian, and provide a tractable method for
estimating this quantity. Experimentally, we show that our method is capable of
detecting when a pre-trained model is underspecified on test data, with
applications to out-of-distribution detection, detecting spurious correlates,
and active learning.
