The Mean-Squared Error of Double Q-Learning
  In this paper, we establish a theoretical comparison between the asymptotic
mean-squared error of Double Q-learning and Q-learning. Our result builds upon
an analysis for linear stochastic approximation based on Lyapunov equations and
applies to both tabular setting and with linear function approximation,
provided that the optimal policy is unique and the algorithms converge. We show
that the asymptotic mean-squared error of Double Q-learning is exactly equal to
that of Q-learning if Double Q-learning uses twice the learning rate of
Q-learning and outputs the average of its two estimators. We also present some
practical implications of this theoretical observation using simulations.
