Visual Exploration of Machine Learning Model Behavior with Hierarchical
  Surrogate Rule Sets
  One of the potential solutions for model interpretation is to train a
surrogate model: a more transparent model that approximates the behavior of the
model to be explained. Typically, classification rules or decision trees are
used due to the intelligibility of their logic-based expressions. However,
decision trees can grow too deep and rule sets can become too large to
approximate a complex model. Unlike paths on a decision tree that must share
ancestor nodes (conditions), rules are more flexible. However, the unstructured
visual representation of rules makes it hard to make inferences across rules.
To address these issues, we present a workflow that includes novel algorithmic
and interactive solutions. First, we present Hierarchical Surrogate Rules
(HSR), an algorithm that generates hierarchical rules based on user-defined
parameters. We also contribute SuRE, a visual analytics (VA) system that
integrates HSR and interactive surrogate rule visualizations. Particularly, we
present a novel feature-aligned tree to overcome the shortcomings of existing
rule visualizations. We evaluate the algorithm in terms of parameter
sensitivity, time performance, and comparison with surrogate decision trees and
find that it scales reasonably well and outperforms decision trees in many
respects. We also evaluate the visualization and the VA system by a usability
study with 24 volunteers and an observational study with 7 domain experts. Our
investigation shows that the participants can use feature-aligned trees to
perform non-trivial tasks with very high accuracy. We also discuss many
interesting observations that can be useful for future research on designing
effective rule-based VA systems.
