You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  The goal of offline reinforcement learning (RL) is to find an optimal policy
given prerecorded trajectories. Many current approaches customize existing
off-policy RL algorithms, especially actor-critic algorithms in which policy
evaluation and improvement are iterated. However, the convergence of such
approaches is not guaranteed due to the use of complex non-linear function
approximation and an intertwined optimization process. By contrast, we propose
a simple baseline algorithm for offline RL that only performs the policy
evaluation step once so that the algorithm does not require complex
stabilization schemes. Since the proposed algorithm is not likely to converge
to an optimal policy, it is an appropriate baseline for actor-critic algorithms
that ought to be outperformed if there is indeed value in iterative
optimization in the offline setting. Surprisingly, we empirically find that the
proposed algorithm exhibits competitive and sometimes even state-of-the-art
performance in a subset of the D4RL offline RL benchmark. This result suggests
that future work is needed to fully exploit the potential advantages of
iterative optimization in order to justify the reduced stability of such
methods.
