Fine-Grained Neural Network Explanation by Identifying Input Features
  with Predictive Information
  One principal approach for illuminating a black-box neural network is feature
attribution, i.e. identifying the importance of input features for the
network's prediction. The predictive information of features is recently
proposed as a proxy for the measure of their importance. So far, the predictive
information is only identified for latent features by placing an information
bottleneck within the network. We propose a method to identify features with
predictive information in the input domain. The method results in fine-grained
identification of input features' information and is agnostic to network
architecture. The core idea of our method is leveraging a bottleneck on the
input that only lets input features associated with predictive latent features
pass through. We compare our method with several feature attribution methods
using mainstream feature attribution evaluation experiments. The code is
publicly available.
