{'fraction_non_alphanumeric': 0.01790830945558739, 'fraction_numerical': 0.0, 'mean_word_length': 5.26457399103139, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Approach to Evaluating Human Characteristics in Agents

When complex cognitive or social characteristics, such as emotions or organisational behaviour, are incorporated into agent frameworks a rich dynamic environment is needed to test and evaluate these modifications. Simple scenarios simply cannot provide the complexity required to test these behaviours. Not only does increased interest in dynamic behaviour of agents require the investigation of new modelling concepts, but it also demands a new perspective on strategies for evaluation, for often there are no objective measures of success for these models. What, for example, is to say that adding emotion to an agent makes it more realistic, or that one model of emotion is superior to another? To answer these and similar questions, statistical techniques borrowed from the social sciences must be employed to obtain meaningful evaluations from a collection of subjective judges. This paper discusses the use and benefits of a commercial multi-player game as a testing environment for models of human behaviour. It describes a project in which the BDI (belief-desire-intention) agent framework has been modified to have more human-like decision-making strategies, the process of selecting a testbed, and the evaluation procedure being applied. The issues that have arisen in this project are then discussed in a more general context.