Delayed Reinforcement Learning by Imitation
When the agent's observations or interactions are delayed, classic
reinforcement learning tools usually fail. In this paper, we propose a simple
yet new and efficient solution to this problem. We assume that, in the
undelayed environment, an efficient policy is known or can be easily learned,
but the task may suffer from delays in practice and we thus want to take them
into account. We present a novel algorithm, Delayed Imitation with Dataset
Aggregation (DIDA), which builds upon imitation learning methods to learn how
to act in a delayed environment from undelayed demonstrations. We provide a
theoretical analysis of the approach that will guide the practical design of
DIDA. These results are also of general interest in the delayed reinforcement
learning literature by providing bounds on the performance between delayed and
undelayed tasks, under smoothness conditions. We show empirically that DIDA
obtains high performances with a remarkable sample efficiency on a variety of
tasks, including robotic locomotion, classic control, and trading.