FADE: FAir Double Ensemble Learning for Observable and Counterfactual
  Outcomes
  Methods for building fair predictors often involve tradeoffs between fairness
and accuracy and between different fairness criteria, but the nature of these
tradeoffs varies. Recent work seeks to characterize these tradeoffs in specific
problem settings, but these methods often do not accommodate users who wish to
improve the fairness of an existing benchmark model without sacrificing
accuracy, or vice versa. These results are also typically restricted to
observable accuracy and fairness criteria. We develop a flexible framework for
fair ensemble learning that allows users to efficiently explore the
fairness-accuracy space or to improve the fairness or accuracy of a benchmark
model. Our framework can simultaneously target multiple observable or
counterfactual fairness criteria, and it enables users to combine a large
number of previously trained and newly trained predictors. We provide
theoretical guarantees that our estimators converge at fast rates. We apply our
method on both simulated and real data, with respect to both observable and
counterfactual accuracy and fairness criteria. We show that, surprisingly,
multiple unfairness measures can sometimes be minimized simultaneously with
little impact on accuracy, relative to unconstrained predictors or existing
benchmark models.
