Analysis of Temporal Difference Learning: Linear System Approach
  The goal of this technical note is to introduce a new finite-time convergence
analysis of temporal difference (TD) learning based on stochastic linear system
models. TD-learning is a fundamental reinforcement learning (RL) to evaluate a
given policy by estimating the corresponding value function for a Markov
decision process. While there has been a series of successful works in
theoretical analysis of TDlearning, it was not until recently that researchers
found some guarantees on its statistical efficiency by developing finite-time
error bounds. In this paper, we propose a simple control theoretic finite-time
analysis of TD-learning, which exploits linear system models and standard
notions in linear system communities. The proposed work provides new simple
templets for RL analysis, and additional insights on TD-learning and RL based
on ideas in control theory.
