Benchmark Evaluation of Counterfactual Algorithms for XAI: From a White
  Box to a Black Box
  Counterfactual explanations have recently been brought to light as a
potentially crucial response to obtaining human-understandable explanations
from predictive models in Explainable Artificial Intelligence (XAI). Despite
the fact that various counterfactual algorithms have been proposed, the state
of the art research still lacks standardised protocols to evaluate the quality
of counterfactual explanations. In this work, we conducted a benchmark
evaluation across different model agnostic counterfactual algorithms in the
literature (DiCE, WatcherCF, prototype, unjustifiedCF), and we investigated the
counterfactual generation process on different types of machine learning models
ranging from a white box (decision tree) to a grey-box (random forest) and a
black box (neural network). We evaluated the different counterfactual
algorithms using several metrics including proximity, interpretability and
functionality for five datasets. The main findings of this work are the
following: (1) without guaranteeing plausibility in the counterfactual
generation process, one cannot have meaningful evaluation results. This means
that all explainable counterfactual algorithms that do not take into
consideration plausibility in their internal mechanisms cannot be evaluated
with the current state of the art evaluation metrics; (2) the counterfactual
generated are not impacted by the different types of machine learning models;
(3) DiCE was the only tested algorithm that was able to generate actionable and
plausible counterfactuals, because it provides mechanisms to constraint
features; (4) WatcherCF and UnjustifiedCF are limited to continuous variables
and can not deal with categorical data.
