On the Generalization Gap in Reparameterizable Reinforcement Learning
  Understanding generalization in reinforcement learning (RL) is a significant
challenge, as many common assumptions of traditional supervised learning theory
do not apply. We focus on the special class of reparameterizable RL problems,
where the trajectory distribution can be decomposed using the reparametrization
trick. For this problem class, estimating the expected return is efficient and
the trajectory can be computed deterministically given peripheral random
variables, which enables us to study reparametrizable RL using supervised
learning and transfer learning theory. Through these relationships, we derive
guarantees on the gap between the expected and empirical return for both
intrinsic and external errors, based on Rademacher complexity as well as the
PAC-Bayes bound. Our bound suggests the generalization capability of
reparameterizable RL is related to multiple factors including "smoothness" of
the environment transition, reward and agent policy function class. We also
empirically verify the relationship between the generalization gap and these
factors through simulations.
