Post-hoc Models for Performance Estimation of Machine Learning Inference
  Estimating how well a machine learning model performs during inference is
critical in a variety of scenarios (for example, to quantify uncertainty, or to
choose from a library of available models). However, the standard accuracy
estimate of softmax confidence is not versatile and cannot reliably predict
different performance metrics (e.g., F1-score, recall) or the performance in
different application scenarios or input domains. In this work, we
systematically generalize performance estimation to a diverse set of metrics
and scenarios and discuss generalized notions of uncertainty calibration. We
propose the use of post-hoc models to accomplish this goal and investigate
design parameters, including the model type, feature engineering, and
performance metric, to achieve the best estimation quality. Emphasis is given
to object detection problems and, unlike prior work, our approach enables the
estimation of per-image metrics such as recall and F1-score. Through extensive
experiments with computer vision models and datasets in three use cases --
mobile edge offloading, model selection, and dataset shift -- we find that
proposed post-hoc models consistently outperform the standard calibrated
confidence baselines. To the best of our knowledge, this is the first work to
develop a unified framework to address different performance estimation
problems for machine learning inference.
