A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning
  Algorithms
  In this paper, we introduce a unified framework for analyzing a large family
of Q-learning algorithms, based on switching system perspectives and ODE-based
stochastic approximation. We show that the nonlinear ODE models associated with
these Q-learning algorithms can be formulated as switched linear systems, and
analyze their asymptotic stability by leveraging existing switching system
theories. Our approach provides the first O.D.E. analysis of the asymptotic
convergence of various Q-learning algorithms, including asynchronous Q-learning
and averaging Q-learning. We also extend the approach to analyze Q-learning
with linear function approximation and derive a new sufficient condition for
its convergence.
