Large-Scale Markov Decision Problems via the Linear Programming Dual
  We consider the problem of controlling a fully specified Markov decision
process (MDP), also known as the planning problem, when the state space is very
large and calculating the optimal policy is intractable. Instead, we pursue the
more modest goal of optimizing over some small family of policies.
Specifically, we show that the family of policies associated with a
low-dimensional approximation of occupancy measures yields a tractable
optimization. Moreover, we propose an efficient algorithm, scaling with the
size of the subspace but not the state space, that is able to find a policy
with low excess loss relative to the best policy in this class. To the best of
our knowledge, such results did not exist in the literature previously. We
bound excess loss in the average cost and discounted cost cases, which are
treated separately. Preliminary experiments show the effectiveness of the
proposed algorithms in a queueing application.
