Human Response to an AI-Based Decision Support System: A User Study on
  the Effects of Accuracy and Bias
  Artificial Intelligence (AI) is increasingly used to build Decision Support
Systems (DSS) across many domains. This paper describes a series of experiments
designed to observe human response to different characteristics of a DSS such
as accuracy and bias, particularly the extent to which participants rely on the
DSS, and the performance they achieve. In our experiments, participants play a
simple online game inspired by so-called "wildcat" (i.e., exploratory) drilling
for oil. The landscape has two layers: a visible layer describing the costs
(terrain), and a hidden layer describing the reward (oil yield). Participants
in the control group play the game without receiving any assistance, while in
treatment groups they are assisted by a DSS suggesting places to drill. For
certain treatments, the DSS does not consider costs, but only rewards, which
introduces a bias that is observable by users. Between subjects, we vary the
accuracy and bias of the DSS, and observe the participants' total score, time
to completion, the extent to which they follow or ignore suggestions. We also
measure the acceptability of the DSS in an exit survey. Our results show that
participants tend to score better with the DSS, that the score increase is due
to users following the DSS advice, and related to the difficulty of the game
and the accuracy of the DSS. We observe that this setting elicits mostly
rational behavior from participants, who place a moderate amount of trust in
the DSS and show neither algorithmic aversion (under-reliance) nor automation
bias (over-reliance).However, their stated willingness to accept the DSS in the
exit survey seems less sensitive to the accuracy of the DSS than their
behavior, suggesting that users are only partially aware of the (lack of)
accuracy of the DSS.
