On the Chattering of SARSA with Linear Function Approximation
  SARSA, a classical on-policy control algorithm for reinforcement learning, is
known to chatter when combined with linear function approximation: SARSA does
not diverge but oscillates in a bounded region. However, little is know about
how fast SARSA converges to that region and how large the region is. In this
paper, we make progress towards solving this open problem by showing the
convergence rate of projected SARSA to a bounded region. Importantly, the
region is much smaller than the ball used for projection provided that the the
magnitude of the reward is not too large. Our analysis applies to expected
SARSA as well as SARSA($\lambda$). Existing works regarding the convergence of
linear SARSA to a fixed point all require the Lipschitz constant of SARSA's
policy improvement operator to be sufficiently small; our analysis instead
applies to arbitrary Lipschitz constants and thus characterizes the behavior of
linear SARSA for a new regime.
