Assessing the communication gap between AI models and healthcare
  professionals: explainability, utility and trust in AI-driven clinical
  decision-making
  This paper contributes with a pragmatic evaluation framework for explainable
Machine Learning (ML) models for clinical decision support. The study revealed
a more nuanced role for ML explanation models, when these are pragmatically
embedded in the clinical context. Despite the general positive attitude of
healthcare professionals (HCPs) towards explanations as a safety and trust
mechanism, for a significant set of participants there were negative effects
associated with confirmation bias, accentuating model over-reliance and
increased effort to interact with the model. Also, contradicting one of its
main intended functions, standard explanatory models showed limited ability to
support a critical understanding of the limitations of the model. However, we
found new significant positive effects which repositions the role of
explanations within a clinical context: these include reduction of automation
bias, addressing ambiguous clinical cases (cases where HCPs were not certain
about their decision) and support of less experienced HCPs in the acquisition
of new domain knowledge.
