Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under
  Data Augmentation
  While agents trained by Reinforcement Learning (RL) can solve increasingly
challenging tasks directly from visual observations, generalizing learned
skills to novel environments remains very challenging. Extensive use of data
augmentation is a promising technique for improving generalization in RL, but
it is often found to decrease sample efficiency and can even lead to
divergence. In this paper, we investigate causes of instability when using data
augmentation in common off-policy RL algorithms. We identify two problems, both
rooted in high-variance Q-targets. Based on our findings, we propose a simple
yet effective technique for stabilizing this class of algorithms under
augmentation. We perform extensive empirical evaluation of image-based RL using
both ConvNets and Vision Transformers (ViT) on a family of benchmarks based on
DeepMind Control Suite, as well as in robotic manipulation tasks. Our method
greatly improves stability and sample efficiency of ConvNets under
augmentation, and achieves generalization results competitive with
state-of-the-art methods for image-based RL in environments with unseen
visuals. We further show that our method scales to RL with ViT-based
architectures, and that data augmentation may be especially important in this
setting.
