Mirror Learning: A Unifying Framework of Policy Optimisation
  Modern deep reinforcement learning (RL) algorithms are motivated by either
the general policy improvement (GPI) or trust-region learning (TRL) frameworks.
However, algorithms that strictly respect these theoretical frameworks have
proven unscalable. Surprisingly, the only known scalable algorithms violate the
GPI/TRL assumptions, e.g. due to required regularisation or other heuristics.
The current explanation of their empirical success is essentially "by analogy":
they are deemed approximate adaptations of theoretically sound methods.
Unfortunately, studies have shown that in practice these algorithms differ
greatly from their conceptual ancestors. In contrast, in this paper we
introduce a novel theoretical framework, named Mirror Learning, which provides
theoretical guarantees to a large class of algorithms, including TRPO and PPO.
While the latter two exploit the flexibility of our framework, GPI and TRL fit
in merely as pathologically restrictive corner cases thereof. This suggests
that the empirical performance of state-of-the-art methods is a direct
consequence of their theoretical properties, rather than of aforementioned
approximate analogies. Mirror learning sets us free to boldly explore novel,
theoretically sound RL algorithms, a thus far uncharted wonderland.
