Regularized Anderson Acceleration for Off-Policy Deep Reinforcement
  Learning
  Model-free deep reinforcement learning (RL) algorithms have been widely used
for a range of complex control tasks. However, slow convergence and sample
inefficiency remain challenging problems in RL, especially when handling
continuous and high-dimensional state spaces. To tackle this problem, we
propose a general acceleration method for model-free, off-policy deep RL
algorithms by drawing the idea underlying regularized Anderson acceleration
(RAA), which is an effective approach to accelerating the solving of fixed
point problems with perturbations. Specifically, we first explain how policy
iteration can be applied directly with Anderson acceleration. Then we extend
RAA to the case of deep RL by introducing a regularization term to control the
impact of perturbation induced by function approximation errors. We further
propose two strategies, i.e., progressive update and adaptive restart, to
enhance the performance. The effectiveness of our method is evaluated on a
variety of benchmark tasks, including Atari 2600 and MuJoCo. Experimental
results show that our approach substantially improves both the learning speed
and final performance of state-of-the-art deep RL algorithms.
