Regularization Guarantees Generalization in Bayesian Reinforcement
  Learning through Algorithmic Stability
  In the Bayesian reinforcement learning (RL) setting, a prior distribution
over the unknown problem parameters -- the rewards and transitions -- is
assumed, and a policy that optimizes the (posterior) expected return is sought.
A common approximation, which has been recently popularized as meta-RL, is to
train the agent on a sample of $N$ problem instances from the prior, with the
hope that for large enough $N$, good generalization behavior to an unseen test
instance will be obtained. In this work, we study generalization in Bayesian RL
under the probably approximately correct (PAC) framework, using the method of
algorithmic stability. Our main contribution is showing that by adding
regularization, the optimal policy becomes stable in an appropriate sense. Most
stability results in the literature build on strong convexity of the
regularized loss -- an approach that is not suitable for RL as Markov decision
processes (MDPs) are not convex. Instead, building on recent results of fast
convergence rates for mirror descent in regularized MDPs, we show that
regularized MDPs satisfy a certain quadratic growth criterion, which is
sufficient to establish stability. This result, which may be of independent
interest, allows us to study the effect of regularization on generalization in
the Bayesian RL setting.
