Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte
  Carlo Method
  The actor-critic RL is widely used in various robotic control tasks. By
viewing the actor-critic RL from the perspective of variational inference (VI),
the policy network is trained to obtain the approximate posterior of actions
given the optimality criteria. However, in practice, the actor-critic RL may
yield suboptimal policy estimates due to the amortization gap and insufficient
exploration. In this work, inspired by the previous use of Hamiltonian Monte
Carlo (HMC) in VI, we propose to integrate the policy network of actor-critic
RL with HMC, which is termed as {\it Hamiltonian Policy}. As such we propose to
evolve actions from the base policy according to HMC, and our proposed method
has many benefits. First, HMC can improve the policy distribution to better
approximate the posterior and hence reduce the amortization gap. Second, HMC
can also guide the exploration more to the regions of action spaces with higher
Q values, enhancing the exploration efficiency. Further, instead of directly
applying HMC into RL, we propose a new leapfrog operator to simulate the
Hamiltonian dynamics. Finally, in safe RL problems, we find that the proposed
method can not only improve the achieved return, but also reduce safety
constraint violations by discarding potentially unsafe actions. With
comprehensive empirical experiments on continuous control baselines, including
MuJoCo and PyBullet Roboschool, we show that the proposed approach is a
data-efficient and easy-to-implement improvement over previous actor-critic
methods.
