Explainability Fact Sheets: A Framework for Systematic Assessment of
  Explainable Approaches
  Explanations in Machine Learning come in many forms, but a consensus
regarding their desired properties is yet to emerge. In this paper we introduce
a taxonomy and a set of descriptors that can be used to characterise and
systematically assess explainable systems along five key dimensions:
functional, operational, usability, safety and validation. In order to design a
comprehensive and representative taxonomy and associated descriptors we
surveyed the eXplainable Artificial Intelligence literature, extracting the
criteria and desiderata that other authors have proposed or implicitly used in
their research. The survey includes papers introducing new explainability
algorithms to see what criteria are used to guide their development and how
these algorithms are evaluated, as well as papers proposing such criteria from
both computer science and social science perspectives. This novel framework
allows to systematically compare and contrast explainability approaches, not
just to better understand their capabilities but also to identify discrepancies
between their theoretical qualities and properties of their implementations. We
developed an operationalisation of the framework in the form of Explainability
Fact Sheets, which enable researchers and practitioners alike to quickly grasp
capabilities and limitations of a particular explainable method. When used as a
Work Sheet, our taxonomy can guide the development of new explainability
approaches by aiding in their critical evaluation along the five proposed
dimensions.
