"What We Can't Measure, We Can't Understand": Challenges to Demographic
  Data Procurement in the Pursuit of Fairness
  As calls for fair and unbiased algorithmic systems increase, so too does the
number of individuals working on algorithmic fairness in industry. However,
these practitioners often do not have access to the demographic data they feel
they need to detect bias in practice. Even with the growing variety of toolkits
and strategies for working towards algorithmic fairness, they almost invariably
require access to demographic attributes or proxies. We investigated this
dilemma through semi-structured interviews with 38 practitioners and
professionals either working in or adjacent to algorithmic fairness.
Participants painted a complex picture of what demographic data availability
and use look like on the ground, ranging from not having access to personal
data of any kind to being legally required to collect and use demographic data
for discrimination assessments. In many domains, demographic data collection
raises a host of difficult questions, including how to balance privacy and
fairness, how to define relevant social categories, how to ensure meaningful
consent, and whether it is appropriate for private companies to infer someone's
demographics. Our research suggests challenges that must be considered by
businesses, regulators, researchers, and community groups in order to enable
practitioners to address algorithmic bias in practice. Critically, we do not
propose that the overall goal of future work should be to simply lower the
barriers to collecting demographic data. Rather, our study surfaces a swath of
normative questions about how, when, and whether this data should be procured,
and, in cases where it is not, what should still be done to mitigate bias.
