Constructing a Good Behavior Basis for Transfer using Generalized Policy
  Updates
  We study the problem of learning a good set of policies, so that when
combined together, they can solve a wide variety of unseen reinforcement
learning tasks with no or very little new data. Specifically, we consider the
framework of generalized policy evaluation and improvement, in which the
rewards for all tasks of interest are assumed to be expressible as a linear
combination of a fixed set of features. We show theoretically that, under
certain assumptions, having access to a specific set of diverse policies, which
we call a set of independent policies, can allow for instantaneously achieving
high-level performance on all possible downstream tasks which are typically
more complex than the ones on which the agent was trained. Based on this
theoretical analysis, we propose a simple algorithm that iteratively constructs
this set of policies. In addition to empirically validating our theoretical
results, we compare our approach with recently proposed diverse policy set
construction methods and show that, while others fail, our approach is able to
build a behavior basis that enables instantaneous transfer to all possible
downstream tasks. We also show empirically that having access to a set of
independent policies can better bootstrap the learning process on downstream
tasks where the new reward function cannot be described as a linear combination
of the features. Finally, we demonstrate how this policy set can be useful in a
lifelong reinforcement learning setting.
