Maximally Invariant Data Perturbation as Explanation
  While several feature scoring methods are proposed to explain the output of
complex machine learning models, most of them lack formal mathematical
definitions. In this study, we propose a novel definition of the feature score
using the maximally invariant data perturbation, which is inspired from the
idea of adversarial example. In adversarial example, one seeks the smallest
data perturbation that changes the model's output. In our proposed approach, we
consider the opposite: we seek the maximally invariant data perturbation that
does not change the model's output. In this way, we can identify important
input features as the ones with small allowable data perturbations. To find the
maximally invariant data perturbation, we formulate the problem as linear
programming. The experiment on the image classification with VGG16 shows that
the proposed method could identify relevant parts of the images effectively.
