A Kernel Loss for Solving the Bellman Equation
  Value function learning plays a central role in many state-of-the-art
reinforcement-learning algorithms. Many popular algorithms like Q-learning do
not optimize any objective function, but are fixed-point iterations of some
variant of Bellman operator that is not necessarily a contraction. As a result,
they may easily lose convergence guarantees, as can be observed in practice. In
this paper, we propose a novel loss function, which can be optimized using
standard gradient-based methods without risking divergence. The key advantage
is that its gradient can be easily approximated using sampled transitions,
avoiding the need for double samples required by prior algorithms like residual
gradient. Our approach may be combined with general function classes such as
neural networks, on either on- or off-policy data, and is shown to work
reliably and effectively in several benchmarks.
