On the Identification of Fair Auditors to Evaluate Recommender Systems
  based on a Novel Non-Comparative Fairness Notion
  Decision-support systems are information systems that offer support to
people's decisions in various applications such as judiciary, real-estate and
banking sectors. Lately, these support systems have been found to be
discriminatory in the context of many practical deployments. In an attempt to
evaluate and mitigate these biases, algorithmic fairness literature has been
nurtured using notions of comparative justice, which relies primarily on
comparing two/more individuals or groups within the society that is supported
by such systems. However, such a fairness notion is not very useful in the
identification of fair auditors who are hired to evaluate latent biases within
decision-support systems. As a solution, we introduce a paradigm shift in
algorithmic fairness via proposing a new fairness notion based on the principle
of non-comparative justice. Assuming that the auditor makes fairness
evaluations based on some (potentially unknown) desired properties of the
decision-support system, the proposed fairness notion compares the system's
outcome with that of the auditor's desired outcome. We show that the proposed
fairness notion also provides guarantees in terms of comparative fairness
notions by proving that any system can be deemed fair from the perspective of
comparative fairness (e.g. individual fairness and statistical parity) if it is
non-comparatively fair with respect to an auditor who has been deemed fair with
respect to the same fairness notions. We also show that the converse holds true
in the context of individual fairness. A brief discussion is also presented
regarding how our fairness notion can be used to identify fair and reliable
auditors, and how we can use them to quantify biases in decision-support
systems.
