Characterizing Fairness Over the Set of Good Models Under Selective
  Labels
  Algorithmic risk assessments are used to inform decisions in a wide variety
of high-stakes settings. Often multiple predictive models deliver similar
overall performance but differ markedly in their predictions for individual
cases, an empirical phenomenon known as the "Rashomon Effect." These models may
have different properties over various groups, and therefore have different
predictive fairness properties. We develop a framework for characterizing
predictive fairness properties over the set of models that deliver similar
overall performance, or "the set of good models." Our framework addresses the
empirically relevant challenge of selectively labelled data in the setting
where the selection decision and outcome are unconfounded given the observed
data features. Our framework can be used to 1) replace an existing model with
one that has better fairness properties; or 2) audit for predictive bias. We
illustrate these uses cases on a real-world credit-scoring task and a
recidivism prediction task.
