Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps
  for Deep Reinforcement Learning
  Saliency maps are frequently used to support explanations of the behavior of
deep reinforcement learning (RL) agents. However, a review of how saliency maps
are used in practice indicates that the derived explanations are often
unfalsifiable and can be highly subjective. We introduce an empirical approach
grounded in counterfactual reasoning to test the hypotheses generated from
saliency maps and assess the degree to which they correspond to the semantics
of RL environments. We use Atari games, a common benchmark for deep RL, to
evaluate three types of saliency maps. Our results show the extent to which
existing claims about Atari games can be evaluated and suggest that saliency
maps are best viewed as an exploratory tool rather than an explanatory tool.
