Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via
  Metagradient
  Exploration-exploitation dilemma has long been a crucial issue in
reinforcement learning. In this paper, we propose a new approach to
automatically balance between these two. Our method is built upon the Soft
Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances
the original task reward and the policy entropy, and hence controls the
trade-off between exploitation and exploration. It is empirically shown that
SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2),
which uses constrained optimization for automatic adjustment, has some
limitations. The core of our method, namely Meta-SAC, is to use metagradient
along with a novel meta objective to automatically tune the entropy temperature
in SAC. We show that Meta-SAC achieves promising performances on several of the
Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most
challenging tasks, humanoid-v2.
