Exploration in Deep Reinforcement Learning: A Comprehensive Survey
  Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning
(MARL) have achieved significant success across a wide range of domains, such
as game AI, autonomous vehicles, robotics and finance. However, DRL and deep
MARL agents are widely known to be sample-inefficient and millions of
interactions are usually needed even for relatively simple game settings, thus
preventing the wide application in real-industry scenarios. One bottleneck
challenge behind is the well-known exploration problem, i.e., how to
efficiently explore the unknown environments and collect informative
experiences that could benefit the policy learning most.
  In this paper, we conduct a comprehensive survey on existing exploration
methods in DRL and deep MARL for the purpose of providing understandings and
insights on the critical problems and solutions. We first identify several key
challenges to achieve efficient exploration, which most of the exploration
methods aim at addressing. Then we provide a systematic survey of existing
approaches by classifying them into two major categories: uncertainty-oriented
exploration and intrinsic motivation-oriented exploration. The essence of
uncertainty-oriented exploration is to leverage the quantification of the
epistemic and aleatoric uncertainty to derive efficient exploration. By
contrast, intrinsic motivation-oriented exploration methods usually incorporate
different reward agnostic information for intrinsic exploration guidance.
Beyond the above two main branches, we also conclude other exploration methods
which adopt sophisticated techniques but are difficult to be classified into
the above two categories. In addition, we provide a comprehensive empirical
comparison of exploration methods for DRL on a set of commonly used benchmarks.
Finally, we summarize the open problems of exploration in DRL and deep MARL and
point out a few future directions.
