Generalised Policy Improvement with Geometric Policy Composition
We introduce a method for policy improvement that interpolates between the
greedy approach of value-based reinforcement learning (RL) and the full
planning approach typical of model-based RL. The new method builds on the
concept of a geometric horizon model (GHM, also known as a gamma-model), which
models the discounted state-visitation distribution of a given policy. We show
that we can evaluate any non-Markov policy that switches between a set of base
Markov policies with fixed probability by a careful composition of the base
policy GHMs, without any additional learning. We can then apply generalised
policy improvement (GPI) to collections of such non-Markov policies to obtain a
new Markov policy that will in general outperform its precursors. We provide a
thorough theoretical analysis of this approach, develop applications to
transfer and standard RL, and empirically demonstrate its effectiveness over
standard GPI on a challenging deep RL continuous control task. We also provide
an analysis of GHM training methods, proving a novel convergence result
regarding previously proposed methods and showing how to train these models
stably in deep RL settings.