Do You Need the Entropy Reward (in Practice)?
  Maximum entropy (MaxEnt) RL maximizes a combination of the original task
reward and an entropy reward. It is believed that the regularization imposed by
entropy, on both policy improvement and policy evaluation, together contributes
to good exploration, training convergence, and robustness of learned policies.
This paper takes a closer look at entropy as an intrinsic reward, by conducting
various ablation studies on soft actor-critic (SAC), a popular representative
of MaxEnt RL. Our findings reveal that in general, entropy rewards should be
applied with caution to policy evaluation. On one hand, the entropy reward,
like any other intrinsic reward, could obscure the main task reward if it is
not properly managed. We identify some failure cases of the entropy reward
especially in episodic Markov decision processes (MDPs), where it could cause
the policy to be overly optimistic or pessimistic. On the other hand, our
large-scale empirical study shows that using entropy regularization alone in
policy improvement, leads to comparable or even better performance and
robustness than using it in both policy improvement and policy evaluation.
Based on these observations, we recommend either normalizing the entropy reward
to a zero mean (SACZero), or simply removing it from policy evaluation
(SACLite) for better practical results.
