Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep
  Classifiers
  Algorithmic fairness is frequently motivated in terms of a trade-off in which
overall performance is decreased so as to improve performance on disadvantaged
groups where the algorithm would otherwise be less accurate. Contrary to this,
we find that applying existing fairness approaches to computer vision improve
fairness by degrading the performance of classifiers across all groups (with
increased degradation on the best performing groups).
  Extending the bias-variance decomposition for classification to fairness, we
theoretically explain why the majority of fairness classifiers designed for low
capacity models should not be used in settings involving high-capacity models,
a scenario common to computer vision. We corroborate this analysis with
extensive experimental support that shows that many of the fairness heuristics
used in computer vision also degrade performance on the most disadvantaged
groups. Building on these insights, we propose an adaptive augmentation
strategy that, uniquely, of all methods tested, improves performance for the
disadvantaged groups.
