A Typology to Explore and Guide Explanatory Interactive Machine Learning
  Recently, more and more eXplanatory Interactive machine Learning (XIL)
methods have been proposed with the goal of extending a model's learning
process by integrating human user supervision on the model's explanations.
These methods were often developed independently, provide different motivations
and stem from different applications. Notably, up to now, there has not been a
comprehensive evaluation of these works. By identifying a common set of basic
modules and providing a thorough discussion of these modules, our work, for the
first time, comes up with a unification of the various methods into a single
typology. This typology can thus be used to categorize existing and future XIL
methods based on the identified modules. Moreover, our work contributes by
surveying six existing XIL methods. In addition to benchmarking these methods
on their overall ability to revise a model, we perform additional benchmarks
regarding wrong reason revision, interaction efficiency, robustness to feedback
quality, and the ability to revise a strongly corrupted model. Apart from
introducing these novel benchmarking tasks, for improved quantitative
evaluations, we further introduce a novel Wrong Reason (\wrnospace) metric
which measures the average wrong reason activation in a model's explanations to
complement a qualitative inspection. In our evaluations, all methods prove to
revise a model successfully. However, we found significant differences between
the methods on individual benchmark tasks, revealing valuable
application-relevant aspects not only for comparing current methods but also to
motivate the necessity of incorporating these benchmarks in the development of
future XIL methods.
