Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement
  Learning
  Training deep reinforcement learning agents on environments with multiple
levels / scenes / conditions from the same task, has become essential for many
applications aiming to achieve generalization and domain transfer from
simulation to the real world. While such a strategy is helpful with
generalization, the use of multiple scenes significantly increases the variance
of samples collected for policy gradient computations. Current methods continue
to view this collection of scenes as a single Markov Decision Process (MDP)
with a common value function; however, we argue that it is better to treat the
collection as a single environment with multiple underlying MDPs. To this end,
we propose a dynamic value estimation (DVE) technique for these multiple-MDP
environments, motivated by the clustering effect observed in the value function
distribution across different scenes. The resulting agent is able to learn a
more accurate and scene-specific value function estimate (and hence the
advantage function), leading to a lower sample variance. Our proposed approach
is simple to accommodate with several existing implementations (like PPO, A3C)
and results in consistent improvements for a range of ProcGen environments and
the AI2-THOR framework based visual navigation task.
