Discovering the Rationale of Decisions: Experiments on Aligning Learning
  and Reasoning
  In AI and law, systems that are designed for decision support should be
explainable when pursuing justice. In order for these systems to be fair and
responsible, they should make correct decisions and make them using a sound and
transparent rationale. In this paper, we introduce a knowledge-driven method
for model-agnostic rationale evaluation using dedicated test cases, similar to
unit-testing in professional software development. We apply this new method in
a set of machine learning experiments aimed at extracting known knowledge
structures from artificial datasets from fictional and non-fictional legal
settings. We show that our method allows us to analyze the rationale of
black-box machine learning systems by assessing which rationale elements are
learned or not. Furthermore, we show that the rationale can be adjusted using
tailor-made training data based on the results of the rationale evaluation.
