Mixture of Step Returns in Bootstrapped DQN
  The concept of utilizing multi-step returns for updating value functions has
been adopted in deep reinforcement learning (DRL) for a number of years.
Updating value functions with different backup lengths provides advantages in
different aspects, including bias and variance of value estimates, convergence
speed, and exploration behavior of the agent. Conventional methods such as
TD-lambda leverage these advantages by using a target value equivalent to an
exponential average of different step returns. Nevertheless, integrating step
returns into a single target sacrifices the diversity of the advantages offered
by different step return targets. To address this issue, we propose Mixture
Bootstrapped DQN (MB-DQN) built on top of bootstrapped DQN, and uses different
backup lengths for different bootstrapped heads. MB-DQN enables heterogeneity
of the target values that is unavailable in approaches relying only on a single
target value. As a result, it is able to maintain the advantages offered by
different backup lengths. In this paper, we first discuss the motivational
insights through a simple maze environment. In order to validate the
effectiveness of MB-DQN, we perform experiments on the Atari 2600 benchmark
environments, and demonstrate the performance improvement of MB-DQN over a
number of baseline methods. We further provide a set of ablation studies to
examine the impacts of different design configurations of MB-DQN.
