The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and
  the xAUC Metric
  Where machine-learned predictive risk scores inform high-stakes decisions,
such as bail and sentencing in criminal justice, fairness has been a serious
concern. Recent work has characterized the disparate impact that such risk
scores can have when used for a binary classification task. This may not
account, however, for the more diverse downstream uses of risk scores and their
non-binary nature. To better account for this, in this paper, we investigate
the fairness of predictive risk scores from the point of view of a bipartite
ranking task, where one seeks to rank positive examples higher than negative
ones. We introduce the xAUC disparity as a metric to assess the disparate
impact of risk scores and define it as the difference in the probabilities of
ranking a random positive example from one protected group above a negative one
from another group and vice versa. We provide a decomposition of bipartite
ranking loss into components that involve the discrepancy and components that
involve pure predictive ability within each group. We use xAUC analysis to
audit predictive risk scores for recidivism prediction, income prediction, and
cardiac arrest prediction, where it describes disparities that are not evident
from simply comparing within-group predictive performance.
