DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement
  Learning
  Offline reinforcement learning algorithms promise to be applicable in
settings where a fixed dataset is available and no new experience can be
acquired. However, such formulation is inevitably offline-data-hungry and, in
practice, collecting a large offline dataset for one specific task over one
specific environment is also costly and laborious. In this paper, we thus 1)
formulate the offline dynamics adaptation by using (source) offline data
collected from another dynamics to relax the requirement for the extensive
(target) offline data, 2) characterize the dynamics shift problem in which
prior offline methods do not scale well, and 3) derive a simple dynamics-aware
reward augmentation (DARA) framework from both model-free and model-based
offline settings. Specifically, DARA emphasizes learning from those source
transition pairs that are adaptive for the target environment and mitigates the
offline dynamics shift by characterizing state-action-next-state pairs instead
of the typical state-action distribution sketched by prior offline RL methods.
The experimental evaluation demonstrates that DARA, by augmenting rewards in
the source offline dataset, can acquire an adaptive policy for the target
environment and yet significantly reduce the requirement of target offline
data. With only modest amounts of target offline data, our performance
consistently outperforms the prior offline RL methods in both simulated and
real-world tasks.
