An Empirical Study of Rich Subgroup Fairness for Machine Learning
  Kearns et al. [2018] recently proposed a notion of rich subgroup fairness
intended to bridge the gap between statistical and individual notions of
fairness. Rich subgroup fairness picks a statistical fairness constraint (say,
equalizing false positive rates across protected groups), but then asks that
this constraint hold over an exponentially or infinitely large collection of
subgroups defined by a class of functions with bounded VC dimension. They give
an algorithm guaranteed to learn subject to this constraint, under the
condition that it has access to oracles for perfectly learning absent a
fairness constraint. In this paper, we undertake an extensive empirical
evaluation of the algorithm of Kearns et al. On four real datasets for which
fairness is a concern, we investigate the basic convergence of the algorithm
when instantiated with fast heuristics in place of learning oracles, measure
the tradeoffs between fairness and accuracy, and compare this approach with the
recent algorithm of Agarwal et al. [2018], which implements weaker and more
traditional marginal fairness constraints defined by individual protected
attributes. We find that in general, the Kearns et al. algorithm converges
quickly, large gains in fairness can be obtained with mild costs to accuracy,
and that optimizing accuracy subject only to marginal fairness leads to
classifiers with substantial subgroup unfairness. We also provide a number of
analyses and visualizations of the dynamics and behavior of the Kearns et al.
algorithm. Overall we find this algorithm to be effective on real data, and
rich subgroup fairness to be a viable notion in practice.
