A Boosting Approach to Reinforcement Learning
  We study efficient algorithms for reinforcement learning in Markov decision
processes whose complexity is independent of the number of states. This
formulation succinctly captures large scale problems, but is also known to be
computationally hard in its general form. Previous approaches attempt to
circumvent the computational hardness by assuming structure in either
transition function or the value function, or by relaxing the solution
guarantee to a local optimality condition.
  We consider the methodology of boosting, borrowed from supervised learning,
for converting weak learners into an accurate policy. The notion of weak
learning we study is that of sampled-based approximate optimization of linear
functions over policies. Under this assumption of weak learnability, we give an
efficient algorithm that is capable of improving the accuracy of such weak
learning methods, till global optimality is reached. We prove sample complexity
and running time bounds on our method, that are polynomial in the natural
parameters of the problem: approximation guarantee, discount factor,
distribution mismatch and number of actions. In particular, our bound does not
depend on the number of states.
  A technical difficulty in applying previous boosting results, is that the
value function over policy space is not convex. We show how to use a non-convex
variant of the Frank-Wolfe method, coupled with recent advances in gradient
boosting that allow incorporating a weak learner with multiplicative
approximation guarantee, to overcome the non-convexity and attain global
convergence.
