Enhanced Scene Specificity with Sparse Dynamic Value Estimation
  Multi-scene reinforcement learning involves training the RL agent across
multiple scenes / levels from the same task, and has become essential for many
generalization applications. However, the inclusion of multiple scenes leads to
an increase in sample variance for policy gradient computations, often
resulting in suboptimal performance with the direct application of traditional
methods (e.g. PPO, A3C). One strategy for variance reduction is to consider
each scene as a distinct Markov decision process (MDP) and learn a joint value
function dependent on both state (s) and MDP (M). However, this is non-trivial
as the agent is usually unaware of the underlying level at train / test times
in multi-scene RL. Recently, Singh et al. [1] tried to address this by
proposing a dynamic value estimation approach that models the true joint value
function distribution as a Gaussian mixture model (GMM). In this paper, we
argue that the error between the true scene-specific value function and the
predicted dynamic estimate can be further reduced by progressively enforcing
sparse cluster assignments once the agent has explored most of the state space.
The resulting agents not only show significant improvements in the final reward
score across a range of OpenAI ProcGen environments, but also exhibit increased
navigation efficiency while completing a game level.
