Policy Improvement via Imitation of Multiple Oracles
  Despite its promise, reinforcement learning's real-world adoption has been
hampered by the need for costly exploration to learn a good policy. Imitation
learning (IL) mitigates this shortcoming by using an oracle policy during
training as a bootstrap to accelerate the learning process. However, in many
practical situations, the learner has access to multiple suboptimal oracles,
which may provide conflicting advice in a state. The existing IL literature
provides a limited treatment of such scenarios. Whereas in the single-oracle
case, the return of the oracle's policy provides an obvious benchmark for the
learner to compete against, neither such a benchmark nor principled ways of
outperforming it are known for the multi-oracle setting. In this paper, we
propose the state-wise maximum of the oracle policies' values as a natural
baseline to resolve conflicting advice from multiple oracles. Using a reduction
of policy optimization to online learning, we introduce a novel IL algorithm
MAMBA, which can provably learn a policy competitive with this benchmark. In
particular, MAMBA optimizes policies by using a gradient estimator in the style
of generalized advantage estimation (GAE). Our theoretical analysis shows that
this design makes MAMBA robust and enables it to outperform the oracle policies
by a larger margin than the IL state of the art, even in the single-oracle
case. In an evaluation against standard policy gradient with GAE and
AggreVaTe(D), we showcase MAMBA's ability to leverage demonstrations both from
a single and from multiple weak oracles, and significantly speed up policy
optimization.
