{'fraction_non_alphanumeric': 0.02600140548137737, 'fraction_numerical': 0.0028109627547434997, 'mean_word_length': 5.245614035087719, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Expert Evaluation of Probabilistic Explanations

We have developed an automatic explanation generation mechanism for an intelligent assistant. Based on a Markov decision processes (MDP), the assistant determines the optimal action for each state, guiding the user in the operation of an industrial plant. When the user makes an error, the assistant generates an explanation composed of 3 main parts: (i) the recommended action in the current situation; (ii) a graphical representation of the process highlighting the relevant variable; (iii) a verbal explanation. To generate the explanations we combine several knowledge sources, including the MDP and a frame system with information of the components, actions and variables in the process. In this paper we present an evaluation of the explanations generated by our system. A panel of experts compared the explanations given by the system to those given by a human, in terms of: coherence, contents, organization, and precision. We consider 15 different explanations for a variety of situations in the power plan domain, and asked 6 different external domain experts to evaluate them. The results are between “good” and “excellent” in average for all the aspects and for all the experts. We consider that these are encouraging results, as we are comparing the explanations generated automatically against those given by an experienced domain expert, a very high standard.