Heterogeneous Calibration: A post-hoc model-agnostic framework for
  improved generalization
  We introduce the notion of heterogeneous calibration that applies a post-hoc
model-agnostic transformation to model outputs for improving AUC performance on
binary classification tasks. We consider overconfident models, whose
performance is significantly better on training vs test data and give intuition
onto why they might under-utilize moderately effective simple patterns in the
data. We refer to these simple patterns as heterogeneous partitions of the
feature space and show theoretically that perfectly calibrating each partition
separately optimizes AUC. This gives a general paradigm of heterogeneous
calibration as a post-hoc procedure by which heterogeneous partitions of the
feature space are identified through tree-based algorithms and post-hoc
calibration techniques are applied to each partition to improve AUC. While the
theoretical optimality of this framework holds for any model, we focus on deep
neural networks (DNNs) and test the simplest instantiation of this paradigm on
a variety of open-source datasets. Experiments demonstrate the effectiveness of
this framework and the future potential for applying higher-performing
partitioning schemes along with more effective calibration techniques.
