Action Candidate Driven Clipped Double Q-learning for Discrete and
  Continuous Action Tasks
  Double Q-learning is a popular reinforcement learning algorithm in Markov
decision process (MDP) problems. Clipped Double Q-learning, as an effective
variant of Double Q-learning, employs the clipped double estimator to
approximate the maximum expected action value. Due to the underestimation bias
of the clipped double estimator, the performance of clipped Double Q-learning
may be degraded in some stochastic environments. In this paper, in order to
reduce the underestimation bias, we propose an action candidate-based clipped
double estimator for Double Q-learning. Specifically, we first select a set of
elite action candidates with high action values from one set of estimators.
Then, among these candidates, we choose the highest valued action from the
other set of estimators. Finally, we use the maximum value in the second set of
estimators to clip the action value of the chosen action in the first set of
estimators and the clipped value is used for approximating the maximum expected
action value. Theoretically, the underestimation bias in our clipped Double
Q-learning decays monotonically as the number of action candidates decreases.
Moreover, the number of action candidates controls the trade-off between the
overestimation and underestimation biases. In addition, we also extend our
clipped Double Q-learning to continuous action tasks via approximating the
elite continuous action candidates. We empirically verify that our algorithm
can more accurately estimate the maximum expected action value on some toy
environments and yield good performance on several benchmark problems.
