Accuracy comparison across face recognition algorithms: Where are we on
  measuring race bias?
  Previous generations of face recognition algorithms differ in accuracy for
images of different races (race bias). Here, we present the possible underlying
factors (data-driven and scenario modeling) and methodological considerations
for assessing race bias in algorithms. We discuss data driven factors (e.g.,
image quality, image population statistics, and algorithm architecture), and
scenario modeling factors that consider the role of the "user" of the algorithm
(e.g., threshold decisions and demographic constraints). To illustrate how
these issues apply, we present data from four face recognition algorithms (a
previous-generation algorithm and three deep convolutional neural networks,
DCNNs) for East Asian and Caucasian faces. First, dataset difficulty affected
both overall recognition accuracy and race bias, such that race bias increased
with item difficulty. Second, for all four algorithms, the degree of bias
varied depending on the identification decision threshold. To achieve equal
false accept rates (FARs), East Asian faces required higher identification
thresholds than Caucasian faces, for all algorithms. Third, demographic
constraints on the formulation of the distributions used in the test, impacted
estimates of algorithm accuracy. We conclude that race bias needs to be
measured for individual applications and we provide a checklist for measuring
this bias in face recognition algorithms.
