{'fraction_non_alphanumeric': 0.015441176470588236, 'fraction_numerical': 0.0, 'mean_word_length': 5.543269230769231, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning from Trajectory-Based Action Preferences

Conventional reinforcement learning algorithms depend on the availability of a numerical feedback signal. In many domains, this is not readily available and, in fact, constitutes an additional parameter of the problem setting. As a consequence, a fair amount of engineering is required in order to find a reasonable configuration of reward signal and algorithm parameters that facilitates an efficient solution of the problem. Preference-based reinforcement learning has been recently proposed to circumvent such problems by replacing the definition of quantitative reward signals with the observation of qualitative preferences over trajectories. In this paper, we propose a novel algorithm which converts preferences over trajectories into preferences over actions, with the goal of directly identifying preferred actions in a state. In addition, we also aim for a reduction of the number and the sensitivity of hyper-parameters, thereby reducing the required amount of experimentation that is needed for arriving at a good solution. The performance of this new approach is evaluated by two common reinforcement learning benchmark problems. The results are promising, showing convergence rates that are comparable to SARSA, but are more robust and obtained with considerably less demand on tuning the setup.