Striving for Simplicity and Performance in Off-Policy DRL: Output
  Normalization and Non-Uniform Sampling
  We aim to develop off-policy DRL algorithms that not only exceed
state-of-the-art performance but are also simple and minimalistic. For standard
continuous control benchmarks, Soft Actor-Critic (SAC), which employs entropy
maximization, currently provides state-of-the-art performance. We first
demonstrate that the entropy term in SAC addresses action saturation due to the
bounded nature of the action spaces, with this insight, we propose a
streamlined algorithm with a simple normalization scheme or with inverted
gradients. We show that both approaches can match SAC's sample efficiency
performance without the need of entropy maximization, we then propose a simple
non-uniform sampling method for selecting transitions from the replay buffer
during training. Extensive experimental results demonstrate that our proposed
sampling scheme leads to state of the art sample efficiency on challenging
continuous control tasks. We combine all of our findings into one simple
algorithm, which we call Streamlined Off Policy with Emphasizing Recent
Experience, for which we provide robust public-domain code.
