Combining Human Predictions with Model Probabilities via Confusion
  Matrices and Calibration
  An increasingly common use case for machine learning models is augmenting the
abilities of human decision makers. For classification tasks where neither the
human or model are perfectly accurate, a key step in obtaining high performance
is combining their individual predictions in a manner that leverages their
relative strengths. In this work, we develop a set of algorithms that combine
the probabilistic output of a model with the class-level output of a human. We
show theoretically that the accuracy of our combination model is driven not
only by the individual human and model accuracies, but also by the model's
confidence. Empirical results on image classification with CIFAR-10 and a
subset of ImageNet demonstrate that such human-model combinations consistently
have higher accuracies than the model or human alone, and that the parameters
of the combination method can be estimated effectively with as few as ten
labeled datapoints.
