Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via
  Higher-Order Influence Functions
  Deep learning models achieve high predictive accuracy across a broad spectrum
of tasks, but rigorously quantifying their predictive uncertainty remains
challenging. Usable estimates of predictive uncertainty should (1) cover the
true prediction targets with high probability, and (2) discriminate between
high- and low-confidence prediction instances. Existing methods for uncertainty
quantification are based predominantly on Bayesian neural networks; these may
fall short of (1) and (2) -- i.e., Bayesian credible intervals do not guarantee
frequentist coverage, and approximate posterior inference undermines
discriminative accuracy. In this paper, we develop the discriminative jackknife
(DJ), a frequentist procedure that utilizes influence functions of a model's
loss functional to construct a jackknife (or leave-one-out) estimator of
predictive confidence intervals. The DJ satisfies (1) and (2), is applicable to
a wide range of deep learning models, is easy to implement, and can be applied
in a post-hoc fashion without interfering with model training or compromising
its accuracy. Experiments demonstrate that DJ performs competitively compared
to existing Bayesian and non-Bayesian regression baselines.
