SOInter: A Novel Deep Energy Based Interpretation Method for Explaining
  Structured Output Models
  We propose a novel interpretation technique to explain the behavior of
structured output models, which learn mappings between an input vector to a set
of output variables simultaneously. Because of the complex relationship between
the computational path of output variables in structured models, a feature can
affect the value of output through other ones. We focus on one of the outputs
as the target and try to find the most important features utilized by the
structured model to decide on the target in each locality of the input space.
In this paper, we assume an arbitrary structured output model is available as a
black box and argue how considering the correlations between output variables
can improve the explanation performance. The goal is to train a function as an
interpreter for the target output variable over the input space. We introduce
an energy-based training process for the interpreter function, which
effectively considers the structural information incorporated into the model to
be explained. The effectiveness of the proposed method is confirmed using a
variety of simulated and real data sets.
