{'fraction_non_alphanumeric': 0.0212624584717608, 'fraction_numerical': 0.0, 'mean_word_length': 5.51948051948052, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comparison of Contextual Bandit Approaches to Human-in-the-Loop Robot Task Completion with Infrequent Feedback

Artificially intelligent assistive agents are playing an increased role in our work and homes. In contrast with currently predominant conversational agents, whose intelligence derives from dialogue trees and external modules, a fully autonomous domestic or workplace robot must carry out more complex reasoning. Such a robot must make good decisions as soon as possible, learn from experience, respond to feedback, and rely on feedback only as much as necessary. In this research, we narrow the focus of a hypothetical robot assistant to a room-tidying task in a simulated domestic environment. Given an item, the robot chooses where to put it among many destinations, then optionally receives feedback from a human operator. We frame the problem as a contextual bandit, a reinforcement learning approach frequently used in Web recommendation systems. We evaluate epsilon-greedy and LinUCB action selection methods under a variety of infrequent feedback scenarios, with several methods for managing the lack of feedback. Our empirical results show that, while early-episode performance and overall accuracy of epsilon-greedy action selection can be improved through learning from no-response feedback and careful management of remembered training episodes, a baseline LinUCB approach outperforms epsilon-greedy action selection in early-episode performance, overall accuracy, and simplicity.