Explainable Machine Learning in Deployment
  Explainable machine learning offers the potential to provide stakeholders
with insights into model behavior by using various methods such as feature
importance scores, counterfactual explanations, or influential training data.
Yet there is little understanding of how organizations use these methods in
practice. This study explores how organizations view and use explainability for
stakeholder consumption. We find that, currently, the majority of deployments
are not for end users affected by the model but rather for machine learning
engineers, who use explainability to debug the model itself. There is thus a
gap between explainability in practice and the goal of transparency, since
explanations primarily serve internal stakeholders rather than external ones.
Our study synthesizes the limitations of current explainability techniques that
hamper their use for end users. To facilitate end user interaction, we develop
a framework for establishing clear goals for explainability. We end by
discussing concerns raised regarding explainability.
