Action Candidate Based Clipped Double Q-learning for Discrete and
  Continuous Action Tasks
  Double Q-learning is a popular reinforcement learning algorithm in Markov
decision process (MDP) problems. Clipped Double Q-learning, as an effective
variant of Double Q-learning, employs the clipped double estimator to
approximate the maximum expected action value. Due to the underestimation bias
of the clipped double estimator, performance of clipped Double Q-learning may
be degraded in some stochastic environments. In this paper, in order to reduce
the underestimation bias, we propose an action candidate based clipped double
estimator for Double Q-learning. Specifically, we first select a set of elite
action candidates with the high action values from one set of estimators. Then,
among these candidates, we choose the highest valued action from the other set
of estimators. Finally, we use the maximum value in the second set of
estimators to clip the action value of the chosen action in the first set of
estimators and the clipped value is used for approximating the maximum expected
action value. Theoretically, the underestimation bias in our clipped Double
Q-learning decays monotonically as the number of the action candidates
decreases. Moreover, the number of action candidates controls the trade-off
between the overestimation and underestimation biases. In addition, we also
extend our clipped Double Q-learning to continuous action tasks via
approximating the elite continuous action candidates. We empirically verify
that our algorithm can more accurately estimate the maximum expected action
value on some toy environments and yield good performance on several benchmark
problems.
