Average Individual Fairness: Algorithms, Generalization and Experiments
  We propose a new family of fairness definitions for classification problems
that combine some of the best properties of both statistical and individual
notions of fairness. We posit not only a distribution over individuals, but
also a distribution over (or collection of) classification tasks. We then ask
that standard statistics (such as error or false positive/negative rates) be
(approximately) equalized across individuals, where the rate is defined as an
expectation over the classification tasks. Because we are no longer averaging
over coarse groups (such as race or gender), this is a semantically meaningful
individual-level constraint. Given a sample of individuals and classification
problems, we design an oracle-efficient algorithm (i.e. one that is given
access to any standard, fairness-free learning heuristic) for the fair
empirical risk minimization task. We also show that given sufficiently many
samples, the ERM solution generalizes in two directions: both to new
individuals, and to new classification tasks, drawn from their corresponding
distributions. Finally we implement our algorithm and empirically verify its
effectiveness.
