Classification-based Approximate Policy Iteration: Experiments and
  Extended Discussions
  Tackling large approximate dynamic programming or reinforcement learning
problems requires methods that can exploit regularities, or intrinsic
structure, of the problem in hand. Most current methods are geared towards
exploiting the regularities of either the value function or the policy. We
introduce a general classification-based approximate policy iteration (CAPI)
framework, which encompasses a large class of algorithms that can exploit
regularities of both the value function and the policy space, depending on what
is advantageous. This framework has two main components: a generic value
function estimator and a classifier that learns a policy based on the estimated
value function. We establish theoretical guarantees for the sample complexity
of CAPI-style algorithms, which allow the policy evaluation step to be
performed by a wide variety of algorithms (including temporal-difference-style
methods), and can handle nonparametric representations of policies. Our bounds
on the estimation error of the performance loss are tighter than existing
results. We also illustrate this approach empirically on several problems,
including a large HIV control task.
