Comparing Human and Machine Bias in Face Recognition
  Much recent research has uncovered and discussed serious concerns of bias in
facial analysis technologies, finding performance disparities between groups of
people based on perceived gender, skin type, lighting condition, etc. These
audits are immensely important and successful at measuring algorithmic bias but
have two major challenges: the audits (1) use facial recognition datasets which
lack quality metadata, like LFW and CelebA, and (2) do not compare their
observed algorithmic bias to the biases of their human alternatives. In this
paper, we release improvements to the LFW and CelebA datasets which will enable
future researchers to obtain measurements of algorithmic bias that are not
tainted by major flaws in the dataset (e.g. identical images appearing in both
the gallery and test set). We also use these new data to develop a series of
challenging facial identification and verification questions that we
administered to various algorithms and a large, balanced sample of human
reviewers. We find that both computer models and human survey participants
perform significantly better at the verification task, generally obtain lower
accuracy rates on dark-skinned or female subjects for both tasks, and obtain
higher accuracy rates when their demographics match that of the question.
Computer models are observed to achieve a higher level of accuracy than the
survey participants on both tasks and exhibit bias to similar degrees as the
human survey participants.
