FairPrep: Promoting Data to a First-Class Citizen in Studies on
  Fairness-Enhancing Interventions
  The importance of incorporating ethics and legal compliance into
machine-assisted decision-making is broadly recognized. Further, several lines
of recent work have argued that critical opportunities for improving data
quality and representativeness, controlling for bias, and allowing humans to
oversee and impact computational processes are missed if we do not consider the
lifecycle stages upstream from model training and deployment. Yet, very little
has been done to date to provide system-level support to data scientists who
wish to develop and deploy responsible machine learning methods. We aim to fill
this gap and present FairPrep, a design and evaluation framework for
fairness-enhancing interventions.
  FairPrep is based on a developer-centered design, and helps data scientists
follow best practices in software engineering and machine learning. As part of
our contribution, we identify shortcomings in existing empirical studies for
analyzing fairness-enhancing interventions. We then show how FairPrep can be
used to measure the impact of sound best practices, such as hyperparameter
tuning and feature scaling. In particular, our results suggest that the high
variability of the outcomes of fairness-enhancing interventions observed in
previous studies is often an artifact of a lack of hyperparameter tuning.
Further, we show that the choice of a data cleaning method can impact the
effectiveness of fairness-enhancing interventions.
