{'fraction_non_alphanumeric': 0.036666503595714495, 'fraction_numerical': 0.00657991292011154, 'mean_word_length': 4.925072463768116, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Explainable Knowledge Graph Embedding: Inference Reconciliation for Knowledge Inferences Supporting Robot Actions

Learned knowledge graph representations supporting robots contain a wealth of domain knowledge that drives robot behavior. However, there does not exist an inference reconciliation framework that expresses how a knowledge graph representation affects a robot's sequential decision making. We use a pedagogical approach to explain the inferences of a learned, black-box knowledge graph representation, a knowledge graph embedding. Our interpretable model, uses a decision tree classifier to locally approximate the predictions of the black-box model, and provides natural language explanations interpretable by non-experts. Results from our algorithmic evaluation affirm our model design choices, and the results of our user studies with non-experts support the need for the proposed inference reconciliation framework. Critically, results from our simulated robot evaluation indicate that our explanations enable non-experts to correct erratic robot behaviors due to nonsensical beliefs within the black-box.


I. INTRODUCTION
Prior work has shown that complex knowledge inferences afforded by learned knowledge graph representations can be used to improve a robot's robustness in ambiguous or unforeseen scenarios. Some examples include tool substitution [1] and interpolating ambiguous end-user commands [2]. However, these learned knowledge graph representations are usually black-boxes that are not interpretable to a non-expert user, who would require an explanation when the robot has erratic behavior due to an incorrect knowledge inference.
Explainable AI Planning (XAIP) seeks to explain an AI's reasoning to humans in sequential decision-making procedures to promote collaboration. In XAIP, inference reconciliation through dialogue with the AI is one method of explaining an AI's reasoning to a user [3]. The growing variety of questions users may ask an AI addressed by prior work include "Why is action a in plan π?", "Why not this other plan π ?", "Why is this policy (action) optimal?", and others. Additionally, prior work has proposed inference reconciliation frameworks that explain how plans, policies, rationales, and scene-graphs can be leveraged to explain an AI's decision making. However, to the best of our knowledge, no existing inference reconciliation framework explains how a knowledge graph representation affects a robot's decision making.
When interacting with a user, a robot may need to justify its action based on semantic knowledge inferences independent of the robot's plan, policy, or the scene. For example, after asking the robot to fetch their coffee, the user might ask the robot "Why are you looking in the 1 Georgia Institute of Technology, Atlanta, GA. Email: {adaruna3, ddas41, chernova}@gatech.edu refrigerator?", to which the robot may reply "Food is stored in the refrigerator, and coffee is a food". Such an explanation not only elucidates the robot's reasoning, but also provides a valuable opportunity for the user to correct the robot's knowledge (e.g., "coffee is stored in the pantry"). The aim of our work is to develop such explanation capabilities, and ultimately improve the robot's reasoning, by introducing a novel type of inference reconciliation of the form, "Why is knowledge inference i, supporting action a, true?".
We introduce an inference reconciliation framework that answers a user's questions about the knowledge inference that supports a robot's action, based on knowledge graphs, XAI, and natural language ( Figure 1). Our framework uses a pedagogical XAI approach to provide explanations to nonexperts about the inferences made by a learned knowledge graph representation. We develop a graph feature model as the interpretable model, using subgraph feature extraction and decision trees. We train the graph feature model to locally approximate the predictions of the learned knowledge graph representation and provide a grounded, natural language explanation for each prediction.
We evaluate our framework across three dimensions: algorithmic performance, user preference of explanations, and robot task performance. In our algorithmic evaluation, we observe statistically significant improvements in classification fidelity over baseline interpretable graph feature models, substantiating our design choices. In our user preference evaluation, we observe that users prefer our explanations as robot responses with statistically significant differences in 73.3% of analyzed interactions. These differences in preferences validate the need for our inference reconciliation framework as it answers "why questions" that are qualitatively different from prior work. Most importantly, in our robot task evaluation, we observe that non-expert feedback prompted by our explanations can effectively improve robot task performance (117% and 33.7% relative improvement in link-prediction and task execution success rate, respectively). The novelties of our work include the inference reconciliation framework as a system, our interpretable graph feature model, and our framework's ability to generate explanations that help nonexperts improve robot task performance. Specifically: 1. We introduce an inference reconciliation framework that answers a novel type of user questions of the form "Why is knowledge inference i, supporting action a, true?" using knowledge graphs; 2. We develop and evaluate a novel graph feature model that outperforms prior work by statistically significant margins on a household knowledge dataset; 3. We showcase a novel application of explanations within XAIP: improving downstream task performance, namely, robot behavior.

II. RELATED WORKS
Our work is motivated by prior research in three areas: Knowledge Graphs in Robotics, Explainable AI Planning, and Knowledge Base Completion.
Knowledge Graphs (KG) are a method to model the properties and interrelations of world entities [4]. Prior works that apply KGs to robotics have demonstrated improved robustness in robot behavior by enabling robots to make complex knowledge inferences. Examples include, substituting failed demonstration actions in plans for executable actions [5], finding objects in alternate locations [6], using alternative tools for tasks [1], inferring conditional object properties [7], and interpolating ambiguous end-user commands [2]. Most of the efforts in modeling KGs for robotics have been focused on developing computational frameworks capable of complex knowledge inferences (e.g., learning KG structure for fact prediction). Such methods have been extensively demonstrated for tasks in which robots are interacting with non-expert users (e.g., households). These two points together motivate our work, which seeks to make a robot's knowledge inferences understandable by non-experts through explanations, such that non-experts can remedy explanations of incorrect knowledge inferences, improving a robot's overall decision-making.
Explainable AI Planning (XAIP) is a focus area of Explainable AI (XAI), with the goal of explaining an AI's reasoning to humans in complex decision-making procedures to foster trust, long-term interaction, and collaboration [3]. Inference reconciliation through dialogue with the AI is one method of explaining an AI's reasoning, motivated by the notion that users have less computational power than sequential decision making systems (e.g., planners). In inference reconciliation through dialogue, user questions about the AI's decision making are answered using explanations [3].
There are a growing variety of questions user's might ask about an AI's planning and representations affecting an AI's sequential decision making that need to be mapped into explanations as question responses. In [8], [9], causal link chains formed by action pre-and post-conditions within plans are used to answer "Why is action a in plan π?". In [10], unmet properties of alternative plans (e.g., constraints) are highlighted to answer "Why not this other plan π'?". In [11], the frequencies with which the current action lead to high-value future states or actions are used to answer "Why is this policy (action) optimal?". In [12], a mapping between user ascribed rationales and world states are used to answer "Why is action a taken in world state s?". In [13], the transformation (excuse) to make Π solvable is used to answer "Why is the sequential decision making problem Π not solvable?". In [14], relevant scene-graph semantic relationships causing plan failure are verbalized to answer "Why did execution of plan π fail?". To the best of our knowledge, no prior work in XAIP has leveraged KGs as part of the sequential decision making representations that need to be mapped into explanations as question responses.
Knowledge Base (Graph) Completion (KBC) seeks to infer missing facts from a knowledge graph using existing facts [15]. In [15] two branches of KBC techniques that have received much attention are surveyed: latent and graph feature models. Latent feature models infer missing facts based on latent features of graph nodes (i.e., embeddings) and graph feature models infer missing facts based on features extracted from observed graph edges (e.g., paths). Latent feature models tend to outperform graph feature models [16]. However, inferences from latent feature models are not interpretable because all embedding values are learned relative to one another and, therefore, dimensions of latent features have no inherent meaning [16]. However, there exist applications where it is desirable to have accurate inferences that are interpretable (e.g., product recommendation).
Prior works have focused on improving the interpretability of embeddings for latent feature models and explainability of inferences. In [17], [18] embedding interpretability for expert users are provided through attention or importance weights over node features with respect to relation features, respectively. In [19], [20], the reasoning behind or reliability of inferences was explained to non-experts in terms of the observed short alternative paths or "crossover interactions" between inferred and given facts, respectively. In [16], the reasoning supporting inferences was explained by learning the most highly correlated alternative paths. In [21], fully grounded explanations about inferences were provided by using expert labels to learn in a semi-supervised manner which "template" explanation (similar to alternative paths) is best suited to explain an inference. We develop a novel graph feature model and compare our approach to each of these prior works in our experiments.

III. METHODOLOGY
The problem of inference reconciliation is grounded in the notion that users typically have less computational ability compared to AI systems, making it difficult for users to understand an AI's solution. One solution entails providing explanations that aid in the user's inferential capabilities [3]. In our work, we consider robotic frameworks in which robot behavior is driven by knowledge inference. In particular, we assume the robot is making decisions based at least in part on a learned knowledge graph representation. To aid user understanding of robot behavior, we introduce an inference reconciliation framework 1 that answers user questions of the form: "Why is knowledge inference i, supporting action a, true?" (e.g. user asks a robot commanded to find a sponge, "Why do you think you will find a sponge in the sink?"). We leverage both knowledge graphs (KG) and interpretability techniques from XAI to provide non-expert users with natural langauage explanations about a robot's knowledge inference supporting an action. As shown in Figure 1, we first gather facts about a robot's task domain, forming a KG, G. We use the KG G to learn a knowledge graph embedding (KGE), Θ, that enables the robot to make complex knowledge inferences. However KGEs are black-box, and lack a mechanism to explain inferences to non-expert users (Section III-A). To make KGEs more interpretable to non-experts, we follow a pedagogical XAI approach [22] to provide explanations about Θ's inferences, using a graph feature model as the interpretable model, Φ. Both G and Θ serve as inputs to Φ, which performs subgraph feature extraction (SFE) and trains a decision tree classifier to locally approximate the predictions of Θ (Section III-B). Given a prediction from Θ, we then use Φ and Θ to extract and ground the explanation in natural language (Section III-C).

A. Knowledge Graph Representation
Knowledge graphs (KGs) are modeled as a graph G composed of individual facts or triples (h, r, t); h and t are the head and tail entities (respectively) for which the relation r holds, e.g., (cup, hasAction, fill) [23], [24], [25]. KGs that model real-world domains are large, sparse, and incomplete. For example, a KG representing a household, while large, only represents a subset of true facts, which are sparse in a space of many potential facts. We adopt knowledge graph embeddings (KGE) for our knowledge graph representation because KGEs are designed for knowledge graphs that are large-scale and sparse [15]. Additionally, KGEs excel at learning the underlying structure of graphs to infer new facts beyond known facts in a graph (i.e. latent feature models in Section II). We build upon the framework in [25], which uses a KGE to represent G.
KGEs are distributed representations that model G in vector space [15], learning a continuous vector representation from a dataset of triples Here y i denotes whether relation r i ∈ R holds between h i , t i ∈ E. Each entity e ∈ E is encoded as a vector v e ∈ R d E , and each relation r ∈ R is encoded as a mapping between vectors W r ∈ R d R , where d E and d R are the dimensions of vectors and mappings 1 Supplementary materials: https://github.com/adaruna3/explainable-kge respectively [15]. The embeddings for E and R are typically learned using a scoring function f (h, r, t) that assigns higher (lower) values to positive (negative) triples [15]. The learning objective is thus to find a set of embeddings Θ = {v e | e ∈ E}, {W r | r ∈ R} that minimizes the loss L D over D. Loss L D can take many forms depending on the KGE representation used, e.g., Negative Log-Likelihood Loss [26].
We make inferences (i.e. fact predictions) in KGEs by completing a transformation in the embedding space. For example, to infer tails {t j | t j ∈ E ∀j} that might complete (h, r, ), the scores f (h, r, t j ) of all j triples are computed, and triples with scores meeting some classification threshold are classified true. Each score f (h, r, t j ) is the resultant of a sequence of high-dimensional geometric transformations between the head entity vector {v h | h ∈ E}, relation mapping {W r | r ∈ R}, and tail entity vectors {v tj | t ∈ E}. Given the complex and relative nature of KGEs, inferences are not inherently interpretable, as discussed in Section II under KBC. With the ultimate objective of providing transparent explanations of KGE inferences to non-experts, we leverage explainability techniques from XAI to explain an inference.

B. Interpretable Model
Our interpretable model Φ locally approximates the inferences (i.e. fact predictions) of the KGE, denoted as Θ such that Φ can provide explanations of Θ predictions. As discussed in Section II, graph feature models use graph features to infer missing facts. We develop a novel interpretable graph feature model Φ that consists of two components: interpretable features δ derived from G and Θ, and an interpretable classifier λ trained on δ to approximate the predictions of Θ. We begin by extracting the features δ derived from a knowledge graph G as in [27], which is formed from a dataset of triples Here, y i denotes whether relation r i ∈ R holds between h i , t i ∈ E. We then train the interpretable classifier λ over the most relevant subsets of these features to infer a triple (i.e. fact) missing from G.
1) Interpretable Knowledge Graph Features: We use Subgraph Feature Extraction (SFE) from [27] to extract interpretable features, δ, from a graph G , which represents a set facts believed true the the KGE Θ. We begin by forming G using the facts in G classified as true by Θ. We also add to G any facts classified as true by Θ, which we form by switching the head h or tail t for a fact (h, r, t) in G with the top K nearest neighbor entities. Neighbors are determined by cosine similarity in the embedding space. We then used SFE to extract our interpretable features, δ, from G instead of G to get a larger set of features classified as true by Θ. SFE uses bi-directional breadth-first search to find all unique relation paths connecting a pair of entities in G . Relation paths are formed from the sequence of relations that are traversed when following a path in G from a head entity h to a tail entity t, where h, t ∈ E. Therefore, for a relation pathP composed of L relations,P represents each relation on a path where ∈ {1...L}. We encode the unique relation paths connecting entities h, t paired by a relation r as features using one-hot encoding.
2) Explainable Model Training: We train a separate interpretable model λ, a decision tree, for each knowledge inference on a subset of available features that maximize λ's classification fidelity to Θ. The joint contributions of relation path combinations extracted from G may not be modeled as linear combinations of individual paths. Therefore, we use a decision tree for our interpretable model, λ, given that decision trees are able to model nonlinear decision boundaries, while remaining interpretable. Additionally, decision trees have more explicit semantics about which relation paths (i.e. features) contribute to a classification (i.e. features along the decision path), excluding extraneous relation paths that may be correlated with a class but are unnecessary to make the classification. The set of features (i.e. relation paths) selected to train a decision tree for each inference are localized to those that maximize the classification fidelity between the decision tree λ and embedding Θ. Given a fact (h, r, t) to be inferred, in which h, t ∈ E and r ∈ R, we implemented the locality by selecting the K nearest neighbor facts (h k , r, t k ), where k ∈ {1...K}, sharing a common relation r such that the classification fidelity between Φ and Θ is maximized. Neighbors are determined by cosine similarity in the embedding space (e.g., cosine(v h , v h k )).

C. Extracting and Grounding Explanations
Given a knowledge inference extracted from a user's "why question" during inference reconciliation, we use the classifier λ in Φ to extract the relevant relation paths, the embedding Θ to ground these relation paths to paths in G , and templates to convert the grounded paths into natural language explanations. The knowledge inference takes the form (h, r, t), where h, t ∈ E and r ∈ R. Assuming Θ and Φ are in agreement about the classification of the query knowledge inference, we extract N relevant relation pathŝ P n , where n ∈ {1...N }, between h and t that were present on the decision path when the decision tree (i.e. λ) performed classification. For each pathP n , we perform bi-directional breadth first search between h and t using relations from P n in order to ground the relation paths using Θ. Therefore, for a relation pathP n composed of L relationsP n , where ∈ {1...L}, the breadth first search at h begins with the relationP 0 n while the backward breath first search at t begins with the relationP L−1 n . Both recursive searches repeatedly perform inference using Θ by classifying the tails (heads) that complete the previous head (tail) and relation inP n for the current search step. When there is overlap between the two searches, an inference is performed that simultaneously classifies the connecting entity serving as a head and tail using Θ to ensure the connecting entity exists for the path. After completing the search, all grounded paths P n made up of relationships classified as true by the Θ have been found and ranked in order of belief according to Θ, which can be accumulated during the search. These grounded paths can then be post-processed using templates for each relation type r ∈ R and automatically corrected for grammar to produce natural-language explanations to users [28].

IV. EXPERIMENTAL EVALUATIONS
We evaluated our inference reconciliation framework with respect to algorithmic performance, user preference, and robot task performance. In our algorithmic evaluation, we used a household robot dataset to compare our interpretable graph feature model Φ with prior work and performed an ablation study on the crucial components of our algorithm design. In our user preference evaluation, we measured whether there were significant differences in user's preferences towards our explanations when a robot is asked "why questions" during task execution. In our robot task evaluation, we measured how non-expert feedback to the robot elicited by our explanations affected robot task performance.

A. Evaluation of Interpretable Graph Feature Model
Our first evaluation qualitatively and quantitatively compared our graph feature model with baseline graph feature models. Qualitatively, we considered the different features necessary for our use case that each baseline model lacked compared with ours. Quantitatively, we measured the extent to which the classifications of each considered graph feature model Φ (i.e. ours and baselines) approximates Θ's classifications (i.e. classification fidelity). Classification fidelity is a proxy measure of whether explanations produced by Φ explain Θ's reasoning [22], [16]. For quantitative comparisons we used an evaluation procedure proposed in [16] for the test split of a dataset D = (h, r, t) i , y i | h i , t i ∈ E, r i ∈ R, y i ∈ {0, 1} , with i ∈ {1...|D|} (see Section IV-A.2). Each y i denotes whether relation r i ∈ R holds between entities h i , t i ∈ E. We checked for significant differences in mean classification fidelity using five-fold cross-validation over D.
1) Qualitative Comparison: We performed a qualitative comparison between graph feature models Φ from Section II to select appropriate baselines for quantitative evaluation. The summary of our qualitative comparisons between all graph feature models Φ is shown in Table I. We did not include SimpleE and ITransF in the quantitative comparison because relative attention weights between relations and entities are not interpretable to non-experts. Additionally, our graph feature model was designed to be embedding agnostic, allowing robotics practitioners to use the current SoTA KGE, eliminating CrossE as a baseline. Rule-Mining (DistMult) was not considered because rule-support cannot provide explanations in cases with no positively correlated relation paths because support does not reason about negative correlations between relation paths. We excluded OxKBC because instead of KG correlations, it uses expert annotations to determine which explanation is best suited for a classification, which may not provide interpretability into the KGE's (i.e. robot's) beliefs. Thus, the only prior method we quantitatively compared against is XKE, as it met all previously mentioned considerations critical to our application.
2) Quantitative Comparison: We compared our approach with XKE [16] quantitatively using an evaluation procedure from [16]. We first generated the inputs held constant during

F1 Fidelity
(µ, σ) SimplE [18] Expert N/A ITransF [17] Expert N/A CrossE [20] ¬Expert N/A DistMult [19] ¬Expert N/A OxKBC [21] ¬Expert N/A XKE [16] ¬Expert   (Table II). We used the TuckER [26] KGE, a recent SoTA KGE model based on the Tucker decomposition, to represent KGE Θ learned from D. We then generated a graph feature model Φ for our approach and XKE using the same inputs, the KGE Θ and dataset D. We then measured the classification fidelity of each Φ to Θ. We measured classification fidelity as F1-Fidelity between Φ and Θ classifications, in which the KGE's classifications served as labels [16]. We checked for significant differences in mean F1-Fidelity across a five-fold cross-validation over D using repeated-measures ANOVA and a post-hoc Tukey's test. Please visit the footnote in Section III for supplementary materials detailing the implementation of Θ and each Φ, the tuning of hyper-parameters, the evaluation dataset, results, and statistical analyses that are omitted here for brevity. Our results in Table I show that there is a statistically significant (p=0.001) improvement in the mean F1-Fidelity between our graph feature model and XKE's.

3) Ablation Study:
We further analyzed our approach by performing an ablation study to understand how each component of our graph feature model contributed to the overall improvement in performance. We toggled two novel parts of our graph feature model not present in XKE: the use of decision trees as Φ and the locality of examples to train Φ. We followed the same procedure as in the previous experiment, the results of which are in Table I. The first ablation, (∀, DT) in Table I, shows a significant (p=0.02) drop in performance from including all available relation paths to train Φ, which is still a decision tree. The fidelity of Φ drops due to the challenge of making an interpretable model approximate the global decision boundary of a black box model [22]. The second ablation, (∀, LR) in Table I, shows a significant (p=0.001) drop in performance from modeling Φ as a linear model instead of a decision tree, in addition to including all available relation paths to train Φ. Here, the fidelity of Φ drops due to the non-linear joint contributions of relation paths, discussed in Section III-B.2.

B. Evaluation of Explanation Preferences
Next, we evaluated our inference reconciliation framework from a non-expert's perspective. We performed a user study to characterize the relationship between different types of "why questions" asked to a robot and a non-expert's preferred types of explanations as responses from the robot. The study evaluated two types of "why questions" asked to discern a robot's actions during cleaning tasks: causal and knowledge inference. Our causal questions were those that inquired about the causal need for an action (e.g. "Why will you move to the sink?"). Our knowledge inference questions were those that inquired about the underlying inferences supporting an action (e.g. "Why do you think you will find a sponge in the sink?) (see Table III). Prompted by the "why question", users selected their preferred explanation from a list provided as possible robot responses. Our null hypothesis was that the type of "why question" asked to a robot would not have a significant affect on the type of explanation from the robot preferred by a non-expert.
1) Study Design: We recruited 50 users from Amazon's Mechanical Turk (AMT) to perform the study, all whom were 18 years or older (M=40.7, SD=10.3). The study was between-subjects, given that each participant was randomly assigned to evaluate only one of two types of "why questions" for each interaction. We counterbalanced for ordering effects by randomizing the question order and robot response type order. Of the 50 participants, 17 participants were filtered out for incorrectly answering any of four filtering questions (not included in results) scattered throughout the study, each which had one sensical and nonsensical response (i.e."..take the scrubber to later slay a fire breathing dragon.."). The remaining 33 participants were used to evaluate 15 robot interactions.
In the study, users watched short videos of robot inter- Knowledge Inference Why Question Causal Why Question Why do you think you will find a washing sponge in a sink?
Why will you move to the sink?
Why do you think a scrubber is used to scrub? Why will you grab the scrubber? Why do you think the act of scrubbing an object will make it clean?
Why will you scrub the kitchen table with the scrubber? Knowledge-graph-based Robot Responses I know that dusting can be done using a wash cloth, a wash cloth is usually used on a stall, a stall can be mopped, and the act of mopping an object will make it clean. Therefore, it is possible that the act of dusting an object will make it clean. I know that a cleaning rag is often in a cabinet, a cabinet often can contain a towel, and a towel is used to wipe. Therefore, it is possible that a cleaning rag is used to wipe.
Causal-link-based Robot Responses I will sponge the table to fulfill the goal of the table being clean. I will move from the table to the sink to later grab the scrubber from the sink.
I will take the scrubber to later be able to scrub the table with the scrubber. I will wipe the table to fulfill the goal of the table being clean. actions within a household environment and were prompted with "why questions" about each interaction. Specifically, in each video, the robot interacted with the world and executed one unique action towards a cleaning task before being interrupted, and the user answered a designated "why question" that was either a causal or knowledge inference question (example provided in Figure 2). Users were then tasked with selecting the "best robot response" to the "why question". Users were provided with two different explanation types as robot responses: causal-link-based and knowledge-graphbased. The causal-link-based explanations were generated from a recent state-of-the-art plan verbalization and explanation method based on causal-link-chaining [9]. The knowledge-graph-based explanations were generated by our inference reconciliation framework in Section III using the dataset D from Table II. Example questions and responses in the assessment are shown in Table III. Note, we do not evaluate explanations using scene-graphs, policies, or rationales (Section II), given that their assumptions do not align with our robot cleaning task).
2) Study Results: We performed a Chi-square test of independence followed by repeated Fisher's exact method measures to analyze user responses aggregated in an individual contingency table for each robot interaction in the assessment. We accounted for Type I error due to Fisher's repeated measures using Simes' alpha correction [30]. In Table  IV, we summarize the statistical analyses for each of the 15 robot interactions analyzed, excluding the filtering question instances. We observed that when asked a knowledge inference "why question", participants select a knowledge-graphbased robot response more often than not in all questions except 5, and that 73.3% of these selections were significant (in bold). Similarly, when asked causal "why questions", participants select a causal-link-based robot response more often than not in all questions, and 73.3% of these differences are significant. Overall, our results indicate that non-experts recognize the qualitative differences in the two types of "why questions", which tend to significantly effect their preferred type of robot response. In other words, there is a need for knowledge-based robot responses given they are better suited for knowledge inference "why questions".

C. Validation of Explanations for Downstream Tasks
In our final experiment we evaluated whether non-experts could use our inference reconciliation framework to improve robot task performance. The robot's task was robust task execution, wherein the robot is provided a demonstrated taskplan recorded in a demonstration environment, and asked to generalize the task-plan to new execution environments. Given that robots cannot be assumed to be error-free, we relaxed the assumption from prior experiments that the robot had gathered a high-quality dataset by including robot perception noise (i.e. the robot had a faulty object and affordance detector). We incorporated robot perception noise into a new datasetD by randomly corrupting 30% of facts for each relation in the dataset D from prior experiments (Table II). As a consequence, the KGEΘ learned from thê D dataset often makes nonsensical knowledge inferences (e.g., (sponge, ObjUsedTo, microwave)) that lead to erratic robot behavior when executing tasks. We validated whether nonsensical facts supporting inferences made byΘ can be revealed to non-experts using explanations generated by our inference reconciliation framework. We hypothesized that if non-experts can accurately recognize and correct nonsensical facts in explanations, then that feedback can be used to improveΘ, and in turn, improve robot behavior.
We performed a user study to measure how well nonexperts can correct nonsensical facts within natural language explanations (i.e. correction accuracy). The explanations were generated by our inference reconciliation framework. Specifically, the interpretable model Φ was trained onD for incorrect classifications of facts in D T e whereΘ and Φ provided the same classification (see Section III). The results of the user study informed a confidence interval for the expected non-expert correction accuracy.
1) Study Design: We recruited 19 participants from AMT, all of whom were 18 years or older (M=33.4, SD=5.9). Of the 19 participants, 1 was filtered out for performing below chance on the practice portion of the study. In the study, users were tasked with identifying and correcting nonsensical facts for a series of knowledge inferences and accompanied explanations. Specifically users traversed through each supporting fact of the explanation and selected the "most correct" fact from a list of options, which included the supporting fact from the explanation, three other facts classified as most true by the KGE that could replace the supporting fact from the explanation, and "None of the above" (example in Figure 3).
We formed a confidence interval for the expected nonexpert correction accuracy because exhaustively recruiting users to evaluate all grounded explanations generated for misclassified examples inD T e was impractical due to the large number of grounded explanations (10,000+). Instead, we randomly assigned each user 16 questions sampled from the large set of grounded explanations. In total, 18 users answered 96 sampled questions. We ensured that each sampled question received three responses (randomly sampled), and combined these responses using majority voting to create 6 meta-users. We determined the necessary population size of meta-users to be 6, by forming a 95% confidence interval with a 5% error margin using the sample standard-deviation of meta-user correction accuracy.
2) Study Results: The mean non-expert correction accuracy was 86.6% with an error margin of 4.1%. We confirmed non-expert correction accuracy samples were normally distributed using Shapiro-Wilk's Test (p=0.83). Additionally, to ensure no grouping bias, we performed a one-way ANOVA to test that there were no significant differences in the mean performances of users whose combined responses formed each meta-user (p=0.72).
3) Improvement of KGE: We characterized the effect of non-expert correction accuracy on improving the KGE by measuring inference performance. Our initial KGE wasΘ learned fromD. We corrected 86.6% of the corrupted facts inD to form a datasetD and learned a new KGE,Θ, fromD. Finally, we measured the difference in inference performance betweenΘ andΘ on our original dataset D (from Table II) for the common KGE task, link-prediction [15].
In short, the link-prediction evaluation task is to rank complete triples from incomplete ones in test splits, i.e., rank heads h given (r, t) or tails t given (h, r). To perform link-prediction, each test triple (h, r, t) is first corrupted by replacing the head (or tail) entity with every other possible entity E. Then all corrupted triples that represent a valid relationship between the corresponding entities are removed to avoid underestimating the embedding performance, known as "filtered setting" [15]. Last, scores are computed for each test triplet and its (remaining) corrupted triplets using the scoring function f (h, r, t), then ranked in order of belief. For each test triple (h, r, t), the mean reciprocal rank of the test triple is calculated as a measure of inference performance. We observed that the mean non-expert correction accuracy provided a 117% relative improvement in MRR betweenΘ andΘ for the link-prediction task. Shown in Figure 4 is the MRR ofΘ as a red dot (32.1%) andΘ as a green dot (69.9%). Figure 4 also includes interpolated results for the range of possible non-expert correction accuracy.
4) Improvement of Robot Behavior: As a final result, we characterized the effect of non-expert correction accuracy on improving the robot's behavior by measuring robot success rate. We considered a real-world scenario in which a household robot performs robust execution of household cleaning tasks, using the system and definitions in [5]. In robust task execution, the robot is provided an execution environment E x and a demonstrated task plan T d that is recorded in a demonstration environment E d , and asked to find a modified task plan T x , executable in E x , that accomplishes the goal(s) of T d . Task plans are defined as a sequence of primitive actions, where each primitive action may or may not be parameterized by objects. Task plans are incrementally modified using knowledge inferences from a KGE with the system defined in [5] to find an executable task plan. Execution environments are sampled by perturbing the object used to demonstrate the task; changing the object's location, type, or both. The perturbations are made in accordance with the action-object-location distributions present in D, ensuring that objects are not placed at implausible locations (e.g., broom inside the toilet) and that the intended generalization is not unreasonable (e.g. cleaning a table with a washingmachine). As a result, the demonstrated task plan often fails due to unsatisfied pre-conditions of primitive actions, and the robot must generalize the demonstrated task plan to formulate the executable task plan.
We measured the improvement in robot success rate provided by the mean non-expert correction accuracy by deployingΘ andΘ on a simulated robot in a simulation household environment performing robust execution of cleaning tasks. We sampled 50 initial demonstrations with 10 executions environments in each, for a total of 500 robot executions and measured the robot's execution success rate when usingΘ andΘ. Our experiment showed a 33.7% relative improvement in the robot's success rate due to the non-expert feedback (i.e. the robot succeeded in 187 sampled environments usingΘ and 250 usingΘ). We measured the success rate of a robot that only repeats the default demonstration and a robot selecting random generalizations of the task plan as reference points for the difficulty of the task, which were 9 and 22 successes, respectively. In addition to the quantitative improvement in success rate, we observed qualitative changes in the robot's generalization behaviors during task execution. Examples are shown in Table V. Each combination of action and object in Table V is an instance of a robot behavior that was corrected by the non-experts. For example, when usingΘ, the robot attempted to dust using a radio. However, by usingΘ, which incorporates the non-expert feedback, those nonsensical task generalizations no longer are attempted.

V. CONCLUSIONS
In summary, we introduce an inference reconciliation framework that answers a user's questions about knowledge inferences supporting robot actions using knowledge graphs, XAI, and natural language. Our framework follows a pedagogical XAI approach, by using an interpetable graph feature model to locally approximate the classifications of a black-box model, a KGE. Through a three-fold evaluation, we demonstrate the importance of our framework both with respect to interpretability as well as improved task performance. Specifically, we show via an algorithmic evaluation that leveraging a decision tree classifier as an interpretable graph feature model in our framework leads to higher F1-Fidelity compared to prior use of linear regression models for explainable KGEs. Additionally, through user evaluations, we demonstrate that our explanations are highly preferred and accessible. Through a user preference evaluation, we demonstrate a significant preference towards our framework's explanations for knowledge inference "why questions." Additionally, when relaxing the assumption that robots are error-free, we showcase the effectiveness of our explanations in helping users identify and correct nonsensical beliefs in robots' knowledge representations, consequently improving robot task performance.