Measurement and Fairness
  We propose measurement modeling from the quantitative social sciences as a
framework for understanding fairness in computational systems. Computational
systems often involve unobservable theoretical constructs, such as
socioeconomic status, teacher effectiveness, and risk of recidivism. Such
constructs cannot be measured directly and must instead be inferred from
measurements of observable properties (and other unobservable theoretical
constructs) thought to be related to them -- i.e., operationalized via a
measurement model. This process, which necessarily involves making assumptions,
introduces the potential for mismatches between the theoretical understanding
of the construct purported to be measured and its operationalization. We argue
that many of the harms discussed in the literature on fairness in computational
systems are direct results of such mismatches. We show how some of these harms
could have been anticipated and, in some cases, mitigated if viewed through the
lens of measurement modeling. To do this, we contribute fairness-oriented
conceptualizations of construct reliability and construct validity that unite
traditions from political science, education, and psychology and provide a set
of tools for making explicit and testing assumptions about constructs and their
operationalizations. We then turn to fairness itself, an essentially contested
construct that has different theoretical understandings in different contexts.
We argue that this contestedness underlies recent debates about fairness
definitions: although these debates appear to be about different
operationalizations, they are, in fact, debates about different theoretical
understandings of fairness. We show how measurement modeling can provide a
framework for getting to the core of these debates.
