Entropy-Augmented Entropy-Regularized Reinforcement Learning and a
  Continuous Path from Policy Gradient to Q-Learning
  Entropy augmented to reward is known to soften the greedy argmax policy to
softmax policy. Entropy augmentation is reformulated and leads to a motivation
to introduce an additional entropy term to the objective function in the form
of KL-divergence to regularize optimization process. It results in a policy
which monotonically improves while interpolating from the current policy to the
softmax greedy policy. This policy is used to build a continuously
parameterized algorithm which optimize policy and Q-function simultaneously and
whose extreme limits correspond to policy gradient and Q-learning,
respectively. Experiments show that there can be a performance gain using an
intermediate algorithm.
