Renewal Monte Carlo: Renewal theory based reinforcement learning
  In this paper, we present an online reinforcement learning algorithm, called
Renewal Monte Carlo (RMC), for infinite horizon Markov decision processes with
a designated start state. RMC is a Monte Carlo algorithm and retains the
advantages of Monte Carlo methods including low bias, simplicity, and ease of
implementation while, at the same time, circumvents their key drawbacks of high
variance and delayed (end of episode) updates. The key ideas behind RMC are as
follows. First, under any reasonable policy, the reward process is ergodic. So,
by renewal theory, the performance of a policy is equal to the ratio of
expected discounted reward to the expected discounted time over a regenerative
cycle. Second, by carefully examining the expression for performance gradient,
we propose a stochastic approximation algorithm that only requires estimates of
the expected discounted reward and discounted time over a regenerative cycle
and their gradients. We propose two unbiased estimators for evaluating
performance gradients---a likelihood ratio based estimator and a simultaneous
perturbation based estimator---and show that for both estimators, RMC converges
to a locally optimal policy. We generalize the RMC algorithm to post-decision
state models and also present a variant that converges faster to an
approximately optimal policy. We conclude by presenting numerical experiments
on a randomly generated MDP, event-triggered communication, and inventory
management.
