Online Learning for Stochastic Shortest Path Model via Posterior
  Sampling
  We consider the problem of online reinforcement learning for the Stochastic
Shortest Path (SSP) problem modeled as an unknown MDP with an absorbing state.
We propose PSRL-SSP, a simple posterior sampling-based reinforcement learning
algorithm for the SSP problem. The algorithm operates in epochs. At the
beginning of each epoch, a sample is drawn from the posterior distribution on
the unknown model dynamics, and the optimal policy with respect to the drawn
sample is followed during that epoch. An epoch completes if either the number
of visits to the goal state in the current epoch exceeds that of the previous
epoch, or the number of visits to any of the state-action pairs is doubled. We
establish a Bayesian regret bound of $O(B_\star S\sqrt{AK})$, where $B_\star$
is an upper bound on the expected cost of the optimal policy, $S$ is the size
of the state space, $A$ is the size of the action space, and $K$ is the number
of episodes. The algorithm only requires the knowledge of the prior
distribution, and has no hyper-parameters to tune. It is the first such
posterior sampling algorithm and outperforms numerically previously proposed
optimism-based algorithms.
