Fairness Sample Complexity and the Case for Human Intervention
  With the aim of building machine learning systems that incorporate standards
of fairness and accountability, we explore explicit subgroup sample complexity
bounds. The work is motivated by the observation that classifier predictions
for real world datasets often demonstrate drastically different metrics, such
as accuracy, when subdivided by specific sensitive variable subgroups. The
reasons for these discrepancies are varied and not limited to the influence of
mitigating variables, institutional bias, underlying population distributions
as well as sampling bias. Among the numerous definitions of fairness that
exist, we argue that at a minimum, principled ML practices should ensure that
classification predictions are able to mirror the underlying sub-population
distributions. However, as the number of sensitive variables increase,
populations meeting at the intersectionality of these variables may simply not
exist or may not be large enough to provide accurate samples for
classification. In these increasingly likely scenarios, we make the case for
human intervention and applying situational and individual definitions of
fairness. In this paper we present lower bounds of subgroup sample complexity
for metric-fair learning based on the theory of Probably Approximately Metric
Fair Learning. We demonstrate that for a classifier to approach a definition of
fairness in terms of specific sensitive variables, adequate subgroup population
samples need to exist and the model dimensionality has to be aligned with
subgroup population distributions. In cases where this is not feasible, we
propose an approach using individual fairness definitions for achieving
alignment. We look at two commonly explored UCI datasets under this lens and
suggest human interventions for data collection for specific subgroups to
achieve approximate individual fairness for linear hypotheses.
