Learning to Control Partially Observed Systems with Finite Memory
  We consider the reinforcement learning problem for partially observed Markov
decision processes (POMDPs) with large or even countably infinite state spaces,
where the controller has access to only noisy observations of the underlying
controlled Markov chain. We consider a natural actor-critic method that employs
a finite internal memory for policy parameterization, and a multi-step temporal
difference learning algorithm for policy evaluation. We establish, to the best
of our knowledge, the first non-asymptotic global convergence of actor-critic
methods for partially observed systems under function approximation. In
particular, in addition to the function approximation and statistical errors
that also arise in MDPs, we explicitly characterize the error due to the use of
finite-state controllers. This additional error is stated in terms of the total
variation distance between the traditional belief state in POMDPs and the
posterior distribution of the hidden state when using a finite-state
controller. Further, we show that this error can be made small in the case of
sliding-block controllers by using larger block sizes.
