Aggressive Q-Learning with Ensembles: Achieving Both High Sample
  Efficiency and High Asymptotic Performance
  Recently, Truncated Quantile Critics (TQC), using distributional
representation of critics, was shown to provide state-of-the-art asymptotic
training performance on all environments from the MuJoCo continuous control
benchmark suite. Also recently, Randomized Ensemble Double Q-Learning (REDQ),
using a high update-to-data ratio and target randomization, was shown to
achieve high sample efficiency that is competitive with state-of-the-art
model-based methods. In this paper, we propose a novel model-free algorithm,
Aggressive Q-Learning with Ensembles (AQE), which improves the
sample-efficiency performance of REDQ and the asymptotic performance of TQC,
thereby providing overall state-of-the-art performance during all stages of
training. Moreover, AQE is very simple, requiring neither distributional
representation of critics nor target randomization.
