Selective Classification Can Magnify Disparities Across Groups
  Selective classification, in which models can abstain on uncertain
predictions, is a natural approach to improving accuracy in settings where
errors are costly but abstentions are manageable. In this paper, we find that
while selective classification can improve average accuracies, it can
simultaneously magnify existing accuracy disparities between various groups
within a population, especially in the presence of spurious correlations. We
observe this behavior consistently across five vision and NLP datasets.
Surprisingly, increasing abstentions can even decrease accuracies on some
groups. To better understand this phenomenon, we study the margin distribution,
which captures the model's confidences over all predictions. For symmetric
margin distributions, we prove that whether selective classification
monotonically improves or worsens accuracy is fully determined by the accuracy
at full coverage (i.e., without any abstentions) and whether the distribution
satisfies a property we call left-log-concavity. Our analysis also shows that
selective classification tends to magnify full-coverage accuracy disparities.
Motivated by our analysis, we train distributionally-robust models that achieve
similar full-coverage accuracies across groups and show that selective
classification uniformly improves each group on these models. Altogether, our
results suggest that selective classification should be used with care and
underscore the importance of training models to perform equally well across
groups at full coverage.
