Teaching Humans When To Defer to a Classifier via Exemplars
  Expert decision makers are starting to rely on data-driven automated agents
to assist them with various tasks. For this collaboration to perform properly,
the human decision maker must have a mental model of when and when not to rely
on the agent. In this work, we aim to ensure that human decision makers learn a
valid mental model of the agent's strengths and weaknesses. To accomplish this
goal, we propose an exemplar-based teaching strategy where humans solve the
task with the help of the agent and try to formulate a set of guidelines of
when and when not to defer. We present a novel parameterization of the human's
mental model of the AI that applies a nearest neighbor rule in local regions
surrounding the teaching examples. Using this model, we derive a near-optimal
strategy for selecting a representative teaching set. We validate the benefits
of our teaching strategy on a multi-hop question answering task using crowd
workers and find that when workers draw the right lessons from the teaching
stage, their task performance improves, we furthermore validate our method on a
set of synthetic experiments.
