Generalization in Reinforcement Learning by Soft Data Augmentation
  Extensive efforts have been made to improve the generalization ability of
Reinforcement Learning (RL) methods via domain randomization and data
augmentation. However, as more factors of variation are introduced during
training, optimization becomes increasingly challenging, and empirically may
result in lower sample efficiency and unstable training. Instead of learning
policies directly from augmented data, we propose SOft Data Augmentation
(SODA), a method that decouples augmentation from policy learning.
Specifically, SODA imposes a soft constraint on the encoder that aims to
maximize the mutual information between latent representations of augmented and
non-augmented data, while the RL optimization process uses strictly
non-augmented data. Empirical evaluations are performed on diverse tasks from
DeepMind Control suite as well as a robotic manipulation task, and we find SODA
to significantly advance sample efficiency, generalization, and stability in
training over state-of-the-art vision-based RL methods.
