PAC Confidence Predictions for Deep Neural Network Classifiers
  A key challenge for deploying deep neural networks (DNNs) in safety critical
settings is the need to provide rigorous ways to quantify their uncertainty. In
this paper, we propose a novel algorithm for constructing predicted
classification confidences for DNNs that comes with provable correctness
guarantees. Our approach uses Clopper-Pearson confidence intervals for the
Binomial distribution in conjunction with the histogram binning approach to
calibrated prediction. In addition, we demonstrate how our predicted
confidences can be used to enable downstream guarantees in two settings: (i)
fast DNN inference, where we demonstrate how to compose a fast but inaccurate
DNN with an accurate but slow DNN in a rigorous way to improve performance
without sacrificing accuracy, and (ii) safe planning, where we guarantee safety
when using a DNN to predict whether a given action is safe based on visual
observations. In our experiments, we demonstrate that our approach can be used
to provide guarantees for state-of-the-art DNNs.
