Composing Entropic Policies using Divergence Correction
  Composing previously mastered skills to solve novel tasks promises dramatic
improvements in the data efficiency of reinforcement learning. Here, we analyze
two recent works composing behaviors represented in the form of action-value
functions and show that they perform poorly in some situations. As part of this
analysis, we extend an important generalization of policy improvement to the
maximum entropy framework and introduce an algorithm for the practical
implementation of successor features in continuous action spaces. Then we
propose a novel approach which addresses the failure cases of prior work and,
in principle, recovers the optimal policy during transfer. This method works by
explicitly learning the (discounted, future) divergence between base policies.
We study this approach in the tabular case and on non-trivial continuous
control problems with compositional structure and show that it outperforms or
matches existing methods across all tasks considered.
