The Impact of Presentation Style on Human-In-The-Loop Detection of
  Algorithmic Bias
  While decision makers have begun to employ machine learning, machine learning
models may make predictions that bias against certain demographic groups.
Semi-automated bias detection tools often present reports of
automatically-detected biases using a recommendation list or visual cues.
However, there is a lack of guidance concerning which presentation style to use
in what scenarios. We conducted a small lab study with 16 participants to
investigate how presentation style might affect user behaviors in reviewing
bias reports. Participants used both a prototype with a recommendation list and
a prototype with visual cues for bias detection. We found that participants
often wanted to investigate the performance measures that were not
automatically detected as biases. Yet, when using the prototype with a
recommendation list, they tended to give less consideration to such measures.
Grounded in the findings, we propose information load and comprehensiveness as
two axes for characterizing bias detection tasks and illustrate how the two
axes could be adopted to reason about when to use a recommendation list or
visual cues.
