Zap Q-Learning With Nonlinear Function Approximation
  Zap Q-learning is a recent class of reinforcement learning algorithms,
motivated primarily as a means to accelerate convergence. Stability theory has
been absent outside of two restrictive classes: the tabular setting, and
optimal stopping. This paper introduces a new framework for analysis of a more
general class of recursive algorithms known as stochastic approximation. Based
on this general theory, it is shown that Zap Q-learning is consistent under a
non-degeneracy assumption, even when the function approximation architecture is
nonlinear. Zap Q-learning with neural network function approximation emerges as
a special case, and is tested on examples from OpenAI Gym. Based on multiple
experiments with a range of neural network sizes, it is found that the new
algorithms converge quickly and are robust to choice of function approximation
architecture.
