The Out-of-Distribution Problem in Explainability and Search Methods for
  Feature Importance Explanations
  Feature importance (FI) estimates are a popular form of explanation, and they
are commonly created and evaluated by computing the change in model confidence
caused by removing certain input features at test time. For example, in the
standard Sufficiency metric, only the top-k most important tokens are kept. In
this paper, we study several under-explored dimensions of FI explanations,
providing conceptual and empirical improvements for this form of explanation.
First, we advance a new argument for why it can be problematic to remove
features from an input when creating or evaluating explanations: the fact that
these counterfactual inputs are out-of-distribution (OOD) to models implies
that the resulting explanations are socially misaligned. The crux of the
problem is that the model prior and random weight initialization influence the
explanations (and explanation metrics) in unintended ways. To resolve this
issue, we propose a simple alteration to the model training process, which
results in more socially aligned explanations and metrics. Second, we compare
among five approaches for removing features from model inputs. We find that
some methods produce more OOD counterfactuals than others, and we make
recommendations for selecting a feature-replacement function. Finally, we
introduce four search-based methods for identifying FI explanations and compare
them to strong baselines, including LIME, Anchors, and Integrated Gradients.
Through experiments with six diverse text classification datasets, we find that
the only method that consistently outperforms random search is a Parallel Local
Search (PLS) that we introduce. Improvements over the second-best method are as
large as 5.4 points for Sufficiency and 17 points for Comprehensiveness. All
supporting code for experiments in this paper is publicly available at
https://github.com/peterbhase/ExplanationSearch.
