On the Fairness of Machine-Assisted Human Decisions
  When machine-learning algorithms are deployed in high-stakes decisions, we
want to ensure that their deployment leads to fair and equitable outcomes. This
concern has motivated a fast-growing literature that focuses on diagnosing and
addressing disparities in machine predictions. However, many machine
predictions are deployed to assist in decisions where a human decision-maker
retains the ultimate decision authority. In this article, we therefore consider
how properties of machine predictions affect the resulting human decisions. We
show in a formal model that the inclusion of a biased human decision-maker can
revert common relationships between the structure of the algorithm and the
qualities of resulting decisions. Specifically, we document that excluding
information about protected groups from the prediction may fail to reduce, and
may even increase, ultimate disparities. While our concrete results rely on
specific assumptions about the data, algorithm, and decision-maker, they show
more broadly that any study of critical properties of complex decision systems,
such as the fairness of machine-assisted human decisions, should go beyond
focusing on the underlying algorithmic predictions in isolation.
