Asymptotics of Reinforcement Learning with Neural Networks
  We prove that a single-layer neural network trained with the Q-learning
algorithm converges in distribution to a random ordinary differential equation
as the size of the model and the number of training steps become large.
Analysis of the limit differential equation shows that it has a unique
stationary solution which is the solution of the Bellman equation, thus giving
the optimal control for the problem. In addition, we study the convergence of
the limit differential equation to the stationary solution. As a by-product of
our analysis, we obtain the limiting behavior of single-layer neural networks
when trained on i.i.d. data with stochastic gradient descent under the
widely-used Xavier initialization.
