{'fraction_non_alphanumeric': 0.015925680159256803, 'fraction_numerical': 0.0, 'mean_word_length': 5.336134453781512, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Balancing Sample Efficiency and Suboptimality in Inverse Reinforcement Learning

We propose a novel formulation for the Inverse Reinforcement Learning (IRL) problem, which jointly accounts for the compatibility with the expert behavior of the identiﬁed reward and its effectiveness for the subsequent forward learning phase. Albeit quite natural, especially when the ﬁnal goal is apprenticeship learning (learn-ing policies from an expert), this aspect has been completely overlooked by IRL approaches so far. We propose a new model-free IRL method that is remarkably able to autonomously ﬁnd a trade-off between the error induced on the learned policy when potentially choosing a sub-optimal reward, and the estimation error caused by using ﬁnite samples in the forward learning phase, which can be controlled by explicitly optimizing also the discount factor of the related learning problem. The approach is based on a min-max formulation for the robust selection of the reward parameters and the discount factor so that the distance between the expert’s policy and the learned policy is minimized in the successive forward learning task when a ﬁnite and possibly small number of samples is available. Differently from the majority of other IRL techniques, our approach does not involve any planning or forward Reinforcement Learning problems to be solved. After presenting the formulation, we provide a numerical scheme for the optimization, and we show its effectiveness on illustrative numerical cases.