Counterfactual Explanation of Brain Activity Classifiers using
  Image-to-Image Transfer by Generative Adversarial Network
  Deep neural networks (DNNs) can accurately decode task-related information
from brain activations. However, because of the nonlinearity of the DNN, the
decisions made by DNNs are hardly interpretable. One of the promising
approaches for explaining such a black-box system is counterfactual
explanation. In this framework, the behavior of a black-box system is explained
by comparing real data and realistic synthetic data that are specifically
generated such that the black-box system outputs an unreal outcome. Here we
introduce a novel generative DNN (counterfactual activation generator, CAG)
that can provide counterfactual explanations for DNN-based classifiers of brain
activations. Importantly, CAG can simultaneously handle image transformation
among multiple classes associated with different behavioral tasks. Using CAG,
we demonstrated counterfactual explanation of DNN-based classifiers that
learned to discriminate brain activations of seven behavioral tasks.
Furthermore, by iterative applications of CAG, we were able to enhance and
extract subtle spatial brain activity patterns that affected the classifier's
decisions. Together, these results demonstrate that the counterfactual
explanation based on image-to-image transformation would be a promising
approach to understand and extend the current application of DNNs in fMRI
analyses.
