{'fraction_non_alphanumeric': 0.026226096006244308, 'fraction_numerical': 0.012228437621959153, 'mean_word_length': 5.267079732594163, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.


Introduction
For clinical Machine Learning (ML), lack of model robustness (Papernot et al., 2016), complexity of clinical modeling tasks (Ghassemi et al., 2018b), and high stakes (Vayena et al., 2018) are some of the technical barriers to practical adoption. Additionally, even a highly accurate ML system is not necessarily sufficient in and of itself to be used and endorsed by clinical staff (Bedoya et al., 2019;Guidi et al., 2015). The uptake and sustained use of ML in healthcare has been more challenging than anticipated, as observed through piloting models for cardiac arrest (Smith et al., 2013) and sepsis (Masino et al., 2019;Elish, 2018). This brings about the question of how truly reliable ML systems can earn clinicians' trust for acceptability.
Understanding ML model behaviour beyond conventional performance metrics has therefore become a necessary component of ML research especially in the healthcare domain. This has led to the development of "interpretable" or "explainable" machine learning models (Gunning, 2017) as a measure to overcome barriers of trust and adoption. Unfortunately, as it stands right now this is a rather ill-defined problem (Lipton, 2016). The ML community has largely resorted to developing novel techniques for model explanations (Guidotti et al., 2018) that are often insightful only to the algorithmic experts (Miller, 2018). Existing work suffers from significant criticism due to the lack of objective definitions of what would validate a model in terms of explainability in the context of clinical practice, while only limited works focus on evaluating the quality and usability of the proposed explanation methods for the target end user (Doshi-Velez and Kim, 2017).
For the purpose of this manuscript, we define "Explainability in ML for Healthcare" as a set of measurable, quantifiable, and transferable attributes associated with an ML system targeted for clinicians. Our goal is to synthesize how explainability is perceived by clinicians and effectively inform on-going research efforts in clinical ML towards building explainable models that cater to end user needs. Additionally, we aim to better understand which "explainability" methods can satisfy clinicians' needs. To this end, we conducted an exploratory survey of 10 clinical stakeholders to: 1) understand if and when 'explainablity' is a useful complement to the clinical ML system; 2) identify the classes of explanation that best serve the end user, emphasizing the context and diversity of their needs; 3) characterize how these considerations can be addressed from an ML standpoint; 4) propose metrics for evaluating clinical explainability methods and identify gaps and research challenges in this area.
Technical Significance We identify when and how explainability can facilitate reliable dissemination of ML model predictions in clinical settings. We curate a set of concrete classes of explanations based on the identified clinicians' needs. We evaluate these in the context of existing explainability literature to highlight gaps and research challenges for clinical ML. By highlighting clinicians' particular needs for explanation as specific technical properties, we hope to channel explainable ML research intended for clinical end use to tackle some of the challenges.
Clinical Relevance Near future will likely see increased adoption of ML solutions in key areas of clinical practice. Building trust between clinicians and ML models, for higher uptake and satisfaction, requires concerted and purposeful efforts from the ML community. This work attempts to pre-emptively identify clinicians' needs, via research surveys, to identify concrete technical challenges that may be addressed using explainable ML with the objective of building reliable ML systems, fostering trust for sustained use aimed at improving clinicians' workflow and practice.

Study Design
We employed an upstream stakeholder engagement method (Corner et al., 2012) to address the aforementioned questions. Upstream engagement here means that the survey is conducted prior to model implementation. We interviewed 10 clinicians in two acute care settings -Intensive Care Unit (ICU) and Emergency Department (ED) with varying years of experience to develop notions of explainability and identify their needs towards building reliable ML systems for their respective clinical practice. Interviews reached saturation (Oreilly and Parker, 2013) wherein no new information pertaining to the main theme of explainability arose during the interviews, after 10 interviews (Hennink et al., 2017). We identify ICU and ED as two sufficiently distinct settings based on their clinical workflow. We developed an interview guide (Jacob and Furgerson, 2012) to address the project goals. Conceptual development of 'explainability' proceeded throughout the interviews (Leech and Onwuegbuzie, 2009) to guide subsequent interview content Jacob and Furgerson (2012). The following describes the upstream stakeholder engagement method and study cohort in detail.

Cohort
We approached a convenience sample of stakeholder clinicians who were familiar with ML based clinical tools and are aware of ongoing developments in this field. Key stakeholders were defined as those who would be end-users of the technology and whose acceptance would ultimately determine successful clinical translation of predictive ML tools. Given the diversity of target clinical tasks, for use in the ICU and ED settings, 6 ICU clinicians and 4 ED clinicians were selected for this exploratory survey. These individuals (5 male, 5 female) were both senior (n = 5, completed residency training) and junior (n = 5, currently in residency training) clinicians to ensure the spectrum of relevant experience was represented among the stakeholders. Discussions were held at the INSTITUTION over a three-week period at clinicians' convenience, and were recorded to facilitate recall accuracy.

Procedures
Clinicians' views were solicited to assess the concept of ML explainability and acceptance by clinicians. Each interview lasted approximately 45-60 minutes. Our interviews began by exploring clinician's notions of 'explainability' to determine what each clinician understood by the term and what he/she expected from ML models in their specific clinical setting. We then introduced the clinician to a hypothetical interactive scenario based on their specialties representing an ML tool incorporated into their division. For the ICU, we introduced a cardiac arrest (CA) prediction tool that reports risk of upcoming cardiac arrest (Tonekaboni et al., 2018). For ED, we introduce a tool that predicts an acuity score based on triage reports (Kwon et al., 2018). We queried the range of possible responses that may arise when considering different aspects of model behavior.

Hypothetical Scenarios
ICU: Suppose a machine learning-based model has been integrated into the EHR system of the hospital. As vital measurements are being constantly recorded in the ICU, this model predicts the probability of a patient having a cardiac arrest within the next few minutes, based on temporal vital measurements. The risk score is updated every few minutes based on the patients current condition. Let us say a patient in your care is being monitored by the model and you receive an alarm that notifies you that they are at imminent risk of a cardiac arrest, but you had not suspected CA for a given patient at the moment.
ED: A machine learning-based model has been integrated into the Emergency Department in your hospital. This system ranks incoming patients based on acuity using the information from the first triage. A patient in your care appears generally well, but the acuity score puts them in a high risk category.
To enhance our understanding of the concept of explainability and trust as perceived by clinicians, we asked the following questions: What would be your first thought upon receiving an alarm like this? How would you go about resolving an alert where your clinical impression differs from the model's prediction? Can you envision a situation where a lab test or other diagnostic tool has told you something that you were not expecting about a patient? How do you think ML models fit in with other diagnostic aids? What kind of information would be helpful to you to interpret this prediction and compare it with your impression about a patient condition? How would you proceed in evaluating the validity of the alarm? Additional questions are detailed in the Appendix.
Ethics Statement Our initial exploratory investigation aligned with the Tri-Council Policy Statement's (TCPS2) (governing research-related activities in Canada) articles 6.11 and 10.1 stipulating activities exempt from Research Ethics Board approval. Activities exempted include: "feasibility of the research, establish research partnerships, or the design of a research proposal".

Results
We organize the results and analyses of the study as follows. We identify concrete and measurable set of explanation classes curated from the qualitative assessments of the surveys (Section 3.1). We identify a few metrics that help assess the utility of any class of explanation towards its clinical utility in Section 3.3. Finally, we summarize how existing explainable ML literature fares in the context of these clinical asks. A full synopsis of the qualitative data including illustrative quotes is depicted in the Appendix.

What makes a model explainable for clinicians?
Throughout the exploratory interviews it was clear that clinicians viewed explainability as a means of justifying their clinical decision-making (for instance, to patients and colleagues) in the context of the model's prediction. To provide these explanations all clinicians expressed the need to understand the clinically relevant model features that align with current evidence-based medical practice. The implemented system/model needs to provide clinicians with information about the context within which the model operates and promote awareness of situations where the model may fall short (e.g., model did not use specific history or did not have information around certain aspect of a patient). Models that fall short in accuracy were deemed acceptable so long as there is clarity around why the model under-performs. While it is ideal to learn models based on as much contextual information, it is not always practical. Furthermore, the complexity of clinical medicine is such that no model is likely to achieve perfect prediction; clinicians, in fact, expect this, and the acknowledgement of this challenge promotes trustworthiness. Clearly specifying features that go into the models for decision making, is a way to facilitate trust in the model as well as directing use in specific patient populations and determining parameters guiding appropriate use. The relevant quote -'the variables that have derived the decision of the model' was brought up by 3 ICU, 1 ED clinicians. This type of transparency is also identified in Mitchell et al. (2018) as being critical and needs to be disclosed to ML model users. While familiar metrics such as reliability, specificity, and sensitivity were important to the initial uptake of an AI tool, a critical factor for continued usage was whether the tool was repeatedly successful in prognosticating their patient's condition in their personal experience. Real-world application was crucial to developing "a sense of when it's working and when it's limited" which meant "alignment with expectations and clinical presentation" [all clinicians].
Clinical thought process for acting on predictions of any assistive tool appears to consist of two primary steps following presentation of the model's prediction: i) understanding, and ii) rationalizing the predictions. Thus classes of explanations for clinical ML models should be designed with the purpose of facilitating the understanding and rationalization process. Clinicians believe that carefully designed visualization and presentation can facilitate further understanding of the model. These features are essential to sustained model use largely due to the immediacy of the clinical picture being captured and in the context of multiple competing attentional demands that require user-friendly visualization. We discerned that there are situations where well-designed visualization is not enough and only additional explanation can facilitate and fill the gap in the identified clinical workflow.

How Does this Translate to Reliable Clinical ML Design?
We determine that a well designed explanation should augment or supplement clinical ML systems to: (c) Reliably disseminate model prediction using task specific representations (e.g. confidence scores).
(d) Provide parsimonious and actionable steps clinicians can undertake. (e.g. potential interventions or data collection).
We further curate the following classes of explanation from qualitative assessments that clinicians identified to most effectively complement model predictions based on our survey. We highlight wherever necessary, the applicability and importance of each explanation class for different settings (ICU vs ED).
Feature Importance: Clinicians repeatedly identified that knowing the subset of features deriving the model outcome, is crucial. This allows them to compare model decision to their clinical judgment, especially in case of a discrepancy. In time-constrained settings such as the emergency department, important features are perceived as a crucial metric to draw the attention of clinicians to specific patient characteristics to determine how to proceed. While multiple clinicians mentioned the need to know relevant variables driving a prediction (see Appendix B), a junior clinician mentioned "you have just a number, you can still use it but in your mind when you put all the variables that make you take a decision, the weight of that variable is going to be less than if you do understand exactly what that number means". It is also important to note that clinicians expect to see patient specific variable importance as well as population level variable importance (James et al., 2013;Tibshirani, 1996). While an extensive survey of feature importance is beyond the scope of this work (see Table 1 for a summary), we highlight that patient-level feature importance is a challenging and far less explored machine learning problem.
Instance Level Explanations: Among commonly researched explainability methods, we investigate whether data instances as explanations (Kim et al., 2016;Koh and Liang, 2017) are useful in any clinical setting. Clinicians view this as finding similar patients (Sun et al., 2012;Zhu et al., 2016;Sharafoddini et al., 2017) and believe that this kind of explanation can be only helpful in specific applications. For example, in cases where an ML model is helping clinicians find a diagnosis for a patient, it is valuable to know the samples the model has previously seen Cai et al. (2019). However, in time constrained settings such as the ICU or ED, clinicians did not find this explanation technique appealing. A nontrivial challenge with this class of explanation, that researchers need to be aware of, is the definition of similarity that is used (Santini and Jain, 1999). Clinicians identified that despite similar outcomes patients may differ significantly in the clinical trajectory to arrive of those outcomes, and vice versa. Various definitions for similarity may be proposed depending on the task the model is performing, and the way the user (clinician) wants to use the explanation. For example, some clinicians were interested in seeing similar samples to their patients because they were curious to know what actions were taken in those cases, and the associated outcomes of the interventions.
Uncertainty: Clinicians overwhelmingly indicated that the model's overall accuracy was not sufficient and its clinical alignment in their judgment often determined their sustained use and trust in the model. That is to say that each time an alert presents a prediction, clinicians will anticipate a clinically significant change in the patient's status that should ideally align with the model's prediction. Previous reports have indicated that despite the determination of expert-agreed threshold for triggering alerts, there are still many alerts not accompanied by a clinically actionable change  which undermines use and endorsement by clinicians (Guidi et al., 2015). This point holds particular importance for clinical ML to ensure developments align with clinicians' expectations to promote trust and sustained model use. Presenting certainty score on model performance or predictions is perceived by clinicians as a sort of explanation that complements the output result. They also suggested that this score can be used as a threshold for reporting results only when they imply that model is very certain of its prediction. Many clinicians noted that "alarm or click fatigue" (indicating the annoyance with repeating response prompts through the EHR system) (Embi and Leonard, 2012) is a significant concern that may be worsened by prediction tools. This issue is ubiqitous across healthcare contexts and requires careful consideration by model developers so as to not perpetuate clinician disillusionment and disengagement with these systems. This makes calibration of complex models (Guo et al., 2017) a significant technical challenge that needs to be addressed for clinical practice.
An additional challenge is the fact that even models performing acceptably on average can have significant individual level errors (Nushi et al., 2018) which is undesirable in clinical practice. There are two sources for uncertainty we highlight (Gal, 2016) that can affect model trust: 1. Model Uncertainty: Uncertainty of model parameters that best explain observed data is known to be a significant source of uncertainty in model performance (Nuzzo, 2014;Zhang et al., 2013). Modeling for this uncertainty can be achieved by actively accounting for distributional differences during training (Gal and Ghahramani, 2016;Schulam and Saria, 2019). Few of these methods have been actively co-opted in clinical healthcare for explainability but nonetheless are crucial to uptake.
In addition, model mis-specification refers to performance deterioration due to a mis-specification of the model class used during training. It can lead to an overall underperforming system and can further reduce the effectiveness of some methods used for dealing with model uncertainty (Wen et al., 2014). This problem is especially under-studied for time-series models that heavily employ deep learning methods (Zhu and Laptev, 2017) and to the best of our knowledge remains mostly unexplored in clinical healthcare employing deep learning methods.
2. Data Uncertainty: This type of uncertainty can come from noisy, missing data or an existing inherent uncertainty in the data. Both of these challenges are commonly faced in clinical ML. Inherently uncertain observation can be for instance a complicated patient with difficult diagnosis, thus is considered a relatively more difficult problem.
Characterizing consistency under missingness has been only briefly studied in supervised classification system (Josse et al., 2019) and needs to be rigorously adopted and evaluated for clinical applications.
Temporal Explanations: 'Patient trajectories that are influential in driving model predictions' was reported to be an important aspect of model explanation. This area remains relatively unexplored in clinical ML explainability literature with a few exceptions (Xu et al., 2018;Choi et al., 2016) focused primarily on attention based deep learning mechanisms (Vaswani et al., 2017). However, explanations based on attention mechanisms can produce inconsistent explanations (Jain and Wallace, 2019). Explainability for temporal data is a relatively challenging task as it requires analyzing higher order temporal dependencies. ML models that perform such predictions, should be able to explain their prediction based on changes in individual patient state. In units such as the ICU, clinicians are interested to see the change of state that has resulted in a certain prediction. Note that this is also related to explanations using Feature Importance (see Sub-section 3.2) as a necessary component of providing explanations in the temporal domain. To the best of our knowledge, not much has been investigated in this domain, as formal definitions of explanations.
Transparent Design Clinicians pointed to the need for models that reflect a similar analytic process to the established methodology of evidence-based medical decision making (Haynes et al., 1997). They anticipate model decisions with similar emphasis on recognizing the clinical features driving the prediction as being more interpretable since it reflects how they currently assess a patient's risk status. Translating a model to a transparent design (e.g. a decision tree) is useful to facilitate rationalization of model behavior, as expressed by one ICU clinician: "would want to know the equation to know what the weights are -but if its variable and if knowing that is too much detail then its just not that helpful."[1 Senior ICU clinician] and "if there is a discrepancy between the current clinical protocol and the model, and we dont know why this is occurring, we are going to be nervous" [1 Senior ED clinician]. To the best of our knowledge, this is the most well studied area of model explanations in general and clinical ML literature. For example, a variety of rule based methods have been proposed precisely for the purpose of transparent clinical design (Lakkaraju et al., 2016;Wang and Rudin, 2015). Model distillation (Hinton et al., 2015) refers to learning a "simpler" model class that perform at par with an complex ML model and can be considered as an attempt toward transparent design (Che et al., 2015). Closely related are regularization based methods ) that attempt to regularize models for enhanced interpretability. These methods tend to assume there are trade-offs between model performance and their perceived "explainability". It is unclear when such a trade-off is acceptable, especially if it amplifies individual level errors. We further argue that such models should be rigorously evaluated for performance under other factors like mis-specification of the distilled model or regularized model, induced biases and general uncertainty to facilitate reliable adoption to clinical practice.

Metrics for Evaluating Explanations
Explanations intended to build trust in clinical settings could benefit from being rigorously evaluated against the following metrics:

Domain Appropriate Representation
The quality of the explanation should be evaluated in terms of whether the representation is coherent with respect to the application task. For instance, for patients in the ICU, clinicians already have a lot of context as to why the patient is currently admitted to this division. Explanations that are redundant to the ICU task are not desirable unless critical to potential clinical workflows. The representation itself should not further obfuscate model behavior for the clinician. This for instance requires careful filtering of the information most useful at any instant (e.g. highlighting "age" even if it is an important feature is not useful if all potential clinical followups are uniform across age or it may not be helpful to receive an alert for a patient who is already en route to receiving a known intervention, like a surgery). The type of explanation will largely be evaluated by rigorous user studies involving stakeholders (Doshi-Velez and Kim, 2017).
Potential Actionability Given a model that satisfies minimal trust based criterion and has been sufficiently evaluated for performance, any complementary explanation should inform follow-up clinical workflow, including rationalization of model prediction (see Section 3.1). This followup can be anything ranging from checking on the patient to ordering additional lab measurements or determining an intervention. Overall, in an applied field like clinical ML, explanations that are informative (in terms of uncerstanding the mode), but have no impact on the workflow are of less importance. Similarly, the explanation should be parsimonious and timely. The nature of explanation should also account for inter-dependant factors that inform its usability. For instance, in an ICU setting, when time to action is extremely limited, counterfactual explanations or patient similarity are not informative, usable, and result in cognitive overload (Henriksen et al., 2008).
Consistency Consistency refers to two factors: (i) The set of explanations should be injective (or changes in model predictions should yield discernable changes in the explanation) (ii) These changes should be invariant to underlying design variations and should only reflect relevant clinical variability. Explanations that are inconsistent across any of these factors effectively violate their reliable actionability, and also negatively impact the trust of clinicians. Consistency is also closely related to robustness of explanations. For instance, such lack of consistency has already been identified using statistical methods for commonly used explanation techniques in deep learning literature (Adebayo et al., 2018;Jain and Wallace, 2019) outside of clinical ML.

What is Missing in Explainable ML?
While a multitute of novel explanability methods have been proposed in the literature (Guidotti et al., 2018), they may not be directly applicable or have to be significantly tailored to clinical settings in the context of asks outlined in Section 3.1. In the following, we focus on identifying a collection of existing state of the art methods than can be used to generate explanations that are usable for clinical stakeholders. We summarize our results in Table 1 and identify possible shortcomings of existing ML approaches for clinical usability and relevance. Note that while a complete review of existing explainability methods is outside the scope of this work, we highlight the most relevant and popular techniques in the context of the classes outlined above. Note that even if an existing explainability method is evaluated in a clinical setting, a rigorous evaluation of these methods for robustness and other inter-dependant factors still necessitates sustained research. Additionally, as noted in Section 4 and Table 1, some methods may work well for specific kinds of data, like images and may have to be nontrivially extended to be applicable to other data, and within the additional complexity of clinical settings.

Related Work
To the best of our knowledge, no prior works have conducted target stakeholder studies for the specific purpose of identifying explainability challenges in clinical ML. In terms of translating ML methods to clinical practice, Escobar et al. (2016);Elish (2018) have extensively recorded their observations in piloting an early sepsis warning system to a single institute over a period of multiple days. A few user studies have evaluated prototypical Electronic Health Recording systems (Mazur et al., 2016) or associated visualization prototypes (Ghassemi et al., 2018a) for efficacy of facilitating diverse clinical workflows (Nolan et al., 2017). Herein we highlight some state of the art literature in explainable Sensitivity Analysis (Saltelli et al., 2008), LRP (Bach et al., 2015) Complex correlation between features of clinical models can be a challenge LIME, Anchors, Shapley Values (Ribeiro et al., 2016(Ribeiro et al., , 2018Lundberg et al., 2018) Further evaluation for consistency required

Instance Level Explanation
Influence functions (Koh and Liang, 2017) Not evaluated on complex clinical models Prototypes and Criticisms (Kim et al., 2016) Limited applicability

Uncertainty
Distributional shift (Subbaswamy and Saria, 2018) Not evaluated on complex clinical models Parameter uncertainty (Gal and Ghahramani, 2016;Schulam and Saria, 2019) Temporal Explanations RETAIN, RAIM (Xu et al., 2018;Choi et al., 2016) Potential lack of consistency due to the attention mechanism (Jain and Wallace, 2019) Transparent Design Rule Based Methods (Lakkaraju et al., 2016;Wang and Rudin, 2015) Less powerful in modeling more complex applications; Generally assume a trade-off of accuracy and explainability ML which we have discussed in terms of their relevance to adoption of ML models in clinical practice in Section 3.1. Explainability in general machine learning has focused on understanding model behavior from different perspectives. Although simpler model classes (Lundberg et al., 2018;Wu et al., 2018) are considered more interpetable by users, a general trade-off between model performance and explainability is assumed (Caruana et al., 2015). Some methods like LIME and Anchors (Ribeiro et al., 2016(Ribeiro et al., , 2018 use simpler model classes to derive feature level explanations (Sundararajan et al., 2017). Abstractions such as data instances (Koh and Liang, 2017;Kim et al., 2016) are used as a means of explaining model behaviors. Visualization is also explored as a method of providing an insight in model behaviour (Yosinski et al., 2015;Ming et al., 2017). We characterize these methods as diagnostic methods as they are not necessarily targeted to provide explanations for the target user but focus on improving model understanding. See Guidotti et al. (2018) for an extensive survey of existing methods for explainable ML in general.
In clinical ML, visualization systems have been evaluated for efficacy of interaction between EHR systems and healthcare practitioners (Ghassemi et al., 2018a). Neural attention mechanisms (Xu et al., 2015) are used for developing interpretable models for heart failure and cataract on-set prediction (Kwon et al., 2019;Choi et al., 2016). Wu et al. (2018) use tree-based regularization for sepsis and HIV treatment, while Wang and Rudin (2015); Ustun and Rudin (2016) build rule based or sparse linear methods for predicting hospital readmissions and sleep apnea scoring systems. Ahmad et al. (2018) conduct the most extensive survey of explainable methods in clinical healthcare, and analyze them in the context of existing notions in general ML and how they relate to clinical healthcare, discussing ethical questions thereof. They conclude that general agreement in the field of clinical healthcare toward explainability is fairly low. Our work attempts to bridge some of this uncertainty by initiating a conversation with and accounting for the needs of stakeholders.
Several works have attempted to codify rigorous evaluation of explainability methods. For instance, Doshi-Velez and Kim (2017) propose ways to evaluate interpretability methods depending on application type and thereby the nature of evaluation, while Narayanan et al. (2018) and Poursabzi-Sangdeh et al. (2018) conduct user studies to understand the quality of many proposed explainability methods. Statistical methods have also been introduced for evaluating the robustness of a few candidate explanation methods like saliency and neural attention architectures (Adebayo et al., 2018;Jain and Wallace, 2019).

Discussion
This work documents the ongoing challenge of translation of clinical ML with a particular focus on explainability through the eyes of end users. We demonstrate how clinicians' views sometimes differ from existing notions of explainability in ML, and propose strategies for enhancing buy-in and trust by focusing on these needs. In light of the objectives highlighted in Section 1, we survey clinicians with diverse specialties and identify when explainability methods can assist in enhancing clinicians' trust in ML models. Our research survey involved creating hypothetical scenarios of deploying a machine learning based predictive tool to carry out specific tasks in the ICU and the ED respectively. We demonstrate that by accounting for target stakeholders, even though the explainability task in clinical settings is significantly diverse, but it can be codified into specific technical challenges. We further outline the need to evaluate clinical explainability methods rigorously under the proposed metrics in Section 3.3. To the best of our knowledge, this is the first attempt at involving ICU and ED stakeholders to identify targeted clinical needs and evaluating it against general machine learning literature in this field. Some of our non-technical observations, as highlighted in the qualitative synopsis in Section 3.1 and Appendix B are corroborated to some extent by the observations of Elish (2018) who followed the development and deployment of a machine learning based sepsis risk detection tool in a clinical setting. Our work however is far more general, as we concretely map conclusions from clinical surveys to prominent gaps in explainable ML literature as it pertains to effective clinical practice. Limitations of our methods are highlighted below.

Limitations
Our research survey for identifying explainability challenges was restricted to ICU and ED specialists. Most specialists we interviewed have reasonable knowledge of clinical ML systems (aware of or involved in academic research involving clinical ML systems). Given the research questions which were focused on prediction tasks, the identified challenges may only have limited applicability to wider class of ML models (for example, reinforcement learning, survival models etc). Applicability of the identified classes of explanations to other clinical settings (like outpatient tasks) needs further evaluation. While our study was an exploration into the research design of a broader study of explainability, we were able to glean useful insights. We are currently preparing an extensive REB protocol to conduct a broader study in this area. A more extensive study along similar lines can overcome the above limitations.
in that the ML systems' value lies in its ability to mimic the clinical acuity of experienced specialists. Perhaps the most important benefit of ML was the ability to facilitate a risk-or acuitybased allocation of attention in the ED. Clinicians in the ED setting particularly described the challenge of not being able to predict with certainty the trajectory of individual patients, and interventions were thus necessarily reactive and subsequent to a negative clinical condition (e.g., blood pressure dropping suddenly). ML should be like having an "constant eye on the patient" [all clinicians] that could support the direction of appropriately timed resources to the bedside of a patient prior to the point of clinical decline. Supporting resource allocation at the precise moment of need was particularly beneficial in the ED, where "there are a thousand things going on at once and you have incomplete information" [all ED clinicians]. In supporting appropriate attention allocation, clinicians refer to the ability of the ML model to provide a quick, reliable piece of information that clinicians can trust to direct them to the right patient at the right time.
Clinical Perception of Explainability of ML Systems: Primarily, clinicians viewed explainability as a means of justifying their clinical decision-making (for instance, to patients and colleagues) in the context of the model's prediction. To them, the term often implied awareness of "the variables that have derived the decision of the model" [3 ICU, 1 ED]. One senior clinician noted, "if it's on a computer then the assumption is that somebody had already figured it out," implying that a lot of features may not require lengthy explanations but instead would require clinical accuracy, as we described in Section 3.1, (Uncertainty). Explanations seemed to be closely tied to visualization and representation of model predictions. A few mentioned it might be helpful to know "parameters that are feeding the model... what data is processed and aggregated" [1 Junior ICU clinician]. Clinicians also noted "if there is a discrepancy between the current clinical protocol and the model, and we dont know why this is occurring, we are going to be nervous" [1 Senior ED clinician]. Universally, awareness of the factors driving the prediction were viewed as essential to determining whether to do additional tests, return to bedside, or intervene: "you have just a number, you can still use it but in your mind when you put all the variables that make you take a decision, the weight of that variable is going to be less than if you do understand exactly what that number means" [1 Junior ICU clinician]. Lastly, for sustained improvement using clinical feedback, knowledge of the model's development was perceived as important insofar as it facilitated the refinement of the tool itself. For example, one clinician noted it is "crucial for clinicians to understand how the model is making predictions to be able to provide feedback on specific conditions that can influence the false positive rate" [1 Junior ED clinician].