Investigating the fidelity of explainable artificial intelligence
  methods for applications of convolutional neural networks in geoscience
  Convolutional neural networks (CNNs) have recently attracted great attention
in geoscience due to their ability to capture non-linear system behavior and
extract predictive spatiotemporal patterns. Given their black-box nature
however, and the importance of prediction explainability, methods of
explainable artificial intelligence (XAI) are gaining popularity as a means to
explain the CNN decision-making strategy. Here, we establish an intercomparison
of some of the most popular XAI methods and investigate their fidelity in
explaining CNN decisions for geoscientific applications. Our goal is to raise
awareness of the theoretical limitations of these methods and gain insight into
the relative strengths and weaknesses to help guide best practices. The
considered XAI methods are first applied to an idealized attribution benchmark,
where the ground truth of explanation of the network is known a priori, to help
objectively assess their performance. Secondly, we apply XAI to a
climate-related prediction setting, namely to explain a CNN that is trained to
predict the number of atmospheric rivers in daily snapshots of climate
simulations. Our results highlight several important issues of XAI methods
(e.g., gradient shattering, inability to distinguish the sign of attribution,
ignorance to zero input) that have previously been overlooked in our field and,
if not considered cautiously, may lead to a distorted picture of the CNN
decision-making strategy. We envision that our analysis will motivate further
investigation into XAI fidelity and will help towards a cautious implementation
of XAI in geoscience, which can lead to further exploitation of CNNs and deep
learning for prediction problems.
