TSEB: More Efficient Thompson Sampling for Policy Learning
  In model-based solution approaches to the problem of learning in an unknown
environment, exploring to learn the model parameters takes a toll on the
regret. The optimal performance with respect to regret or PAC bounds is
achievable, if the algorithm exploits with respect to reward or explores with
respect to the model parameters, respectively. In this paper, we propose TSEB,
a Thompson Sampling based algorithm with adaptive exploration bonus that aims
to solve the problem with tighter PAC guarantees, while being cautious on the
regret as well. The proposed approach maintains distributions over the model
parameters which are successively refined with more experience. At any given
time, the agent solves a model sampled from this distribution, and the sampled
reward distribution is skewed by an exploration bonus in order to generate more
informative exploration. The policy by solving is then used for generating more
experience that helps in updating the posterior over the model parameters. We
provide a detailed analysis of the PAC guarantees, and convergence of the
proposed approach. We show that our adaptive exploration bonus encourages the
additional exploration required for better PAC bounds on the algorithm. We
provide empirical analysis on two different simulated domains.
