The Least Restriction for Offline Reinforcement Learning
  Many practical applications of reinforcement learning (RL) constrain the
agent to learn from a fixed offline dataset of logged interactions, which has
already been gathered, without offering further possibility for data
collection. However, commonly used off-policy RL algorithms, such as the Deep Q
Network and the Deep Deterministic Policy Gradient, are incapable of learning
without data correlated to the distribution under the current policy, making
them ineffective for this offline setting. As the first step towards useful
offline RL algorithms, we analysis the reason of instability in standard
off-policy RL algorithms. It is due to the bootstrapping error. The key to
avoiding this error, is ensuring that the agent's action space does not go out
of the fixed offline dataset. Based on our consideration, a creative offline RL
framework, the Least Restriction (LR), is proposed in this paper. The LR
regards selecting an action as taking a sample from the probability
distribution. It merely set a little limit for action selection, which not only
avoid the action being out of the offline dataset but also remove all the
unreasonable restrictions in earlier approaches (e.g. Batch-Constrained Deep
Q-Learning). In the further, we will demonstrate that the LR, is able to learn
robustly from different offline datasets, including random and suboptimal
demonstrations, on a range of practical control tasks.
