SCORE: Spurious COrrelation REduction for Offline Reinforcement Learning
  Offline reinforcement learning (RL) aims to learn the optimal policy from a
pre-collected dataset without online interactions. Most of the existing studies
focus on distributional shift caused by out-of-distribution actions. However,
even in-distribution actions can raise serious problems. Since the dataset only
contains limited information about the underlying model, offline RL is
vulnerable to spurious correlations, i.e., the agent tends to prefer actions
that by chance lead to high returns, resulting in a highly suboptimal policy.
To address such a challenge, we propose a practical and theoretically
guaranteed algorithm SCORE that reduces spurious correlations by combing an
uncertainty penalty into policy evaluation. We show that this is consistent
with the pessimism principle studied in theory, and the proposed algorithm
converges to the optimal policy with a sublinear rate under mild assumptions.
By conducting extensive experiments on existing benchmarks, we show that SCORE
not only benefits from a solid theory but also obtains strong empirical results
on a variety of tasks.
