Visualization Guidelines for Model Performance Communication Between
  Data Scientists and Subject Matter Experts
  Presenting the complexities of a model's performance is a communication
bottleneck that threatens collaborations between data scientists and subject
matter experts. Accuracy and error metrics alone fail to tell the whole story
of a model - its risks, strengths, and limitations - making it difficult for
subject matter experts to feel confident in deciding to use a model. As a
result, models may fail in unexpected ways if their weaknesses are not clearly
understood. Alternatively, models may go unused, as subject matter experts
disregard poorly presented models in favor of familiar, yet arguably
substandard methods. In this paper, we propose effective use of visualization
as a medium for communication between data scientists and subject matter
experts. Our research addresses the gap between common practices in model
performance communication and the understanding of subject matter experts and
decision makers. We derive a set of communication guidelines and recommended
visualizations for communicating model performance based on interviews of both
data scientists and subject matter experts at the same organization. We conduct
a follow-up study with subject matter experts to evaluate the efficacy of our
guidelines in presentations of model performance with and without our
recommendations. We find that our proposed guidelines made subject matter
experts more aware of the tradeoffs of the presented model. Participants
realized that current communication methods left them without a robust
understanding of the model's performance, potentially giving them misplaced
confidence in the use of the model.
