The Role of Explainability in Assuring Safety of Machine Learning in
  Healthcare
  Established approaches to assuring safety-critical systems and software are
difficult to apply to systems employing ML where there is no clear, pre-defined
specification against which to assess validity. This problem is exacerbated by
the "opaque" nature of ML where the learnt model is not amenable to human
scrutiny. Explainable AI (XAI) methods have been proposed to tackle this issue
by producing human-interpretable representations of ML models which can help
users to gain confidence and build trust in the ML system. However, little work
explicitly investigates the role of explainability for safety assurance in the
context of ML development. This paper identifies ways in which XAI methods can
contribute to safety assurance of ML-based systems. It then uses a concrete
ML-based clinical decision support system, concerning weaning of patients from
mechanical ventilation, to demonstrate how XAI methods can be employed to
produce evidence to support safety assurance. The results are also represented
in a safety argument to show where, and in what way, XAI methods can contribute
to a safety case. Overall, we conclude that XAI methods have a valuable role in
safety assurance of ML-based systems in healthcare but that they are not
sufficient in themselves to assure safety.
