Geometric Value Iteration: Dynamic Error-Aware KL Regularization for
  Reinforcement Learning
  The recent boom in the literature on entropy-regularized reinforcement
learning (RL) approaches reveals that Kullback-Leibler (KL) regularization
brings advantages to RL algorithms by canceling out errors under mild
assumptions. However, existing analyses focus on fixed regularization with a
constant weighting coefficient and do not consider cases where the coefficient
is allowed to change dynamically. In this paper, we study the dynamic
coefficient scheme and present the first asymptotic error bound. Based on the
dynamic coefficient error bound, we propose an effective scheme to tune the
coefficient according to the magnitude of error in favor of more robust
learning. Complementing this development, we propose a novel algorithm,
Geometric Value Iteration (GVI), that features a dynamic error-aware KL
coefficient design with the aim of mitigating the impact of errors on
performance. Our experiments demonstrate that GVI can effectively exploit the
trade-off between learning speed and robustness over uniform averaging of a
constant KL coefficient. The combination of GVI and deep networks shows stable
learning behavior even in the absence of a target network, where algorithms
with a constant KL coefficient would greatly oscillate or even fail to
converge.
