Deep Reinforcement Learning-based UAV Navigation and Control: A Soft
  Actor-Critic with Hindsight Experience Replay Approach
  In this paper, we propose SACHER (soft actor-critic (SAC) with hindsight
experience replay (HER)), which constitutes a class of deep reinforcement
learning (DRL) algorithms. SAC is known as an off-policy model-free DRL
algorithm based on the maximum entropy framework, which outperforms earlier DRL
algorithms in terms of exploration, robustness and learning performance.
However, in SAC, maximizing the entropy-augmented objective may degrade the
optimality of learning outcomes. HER is known as a sample-efficient replay
method that enhances the performance of off-policy DRL algorithms by allowing
the agent to learn from both failures and successes. We apply HER to SAC and
propose SACHER to improve the learning performance of SAC. More precisely,
SACHER achieves the desired optimal outcomes faster and more accurately than
SAC, since HER improves the sample efficiency of SAC. We apply SACHER to the
navigation and control problem of unmanned aerial vehicles (UAVs), where SACHER
generates the optimal navigation path of the UAV under various obstacles in
operation. Specifically, we show the effectiveness of SACHER in terms of the
tracking error and cumulative reward in UAV operation by comparing them with
those of state-of-the-art DRL algorithms, SAC and DDPG. Note that SACHER in UAV
navigation and control problems can be applied to arbitrary models of UAVs.
