Homotopy Based Reinforcement Learning with Maximum Entropy for
  Autonomous Air Combat
  The Intelligent decision of the unmanned combat aerial vehicle (UCAV) has
long been a challenging problem. The conventional search method can hardly
satisfy the real-time demand during high dynamics air combat scenarios. The
reinforcement learning (RL) method can significantly shorten the decision time
via using neural networks. However, the sparse reward problem limits its
convergence speed and the artificial prior experience reward can easily deviate
its optimal convergent direction of the original task, which raises great
difficulties for the RL air combat application. In this paper, we propose a
homotopy-based soft actor-critic method (HSAC) which focuses on addressing
these problems via following the homotopy path between the original task with
sparse reward and the auxiliary task with artificial prior experience reward.
The convergence and the feasibility of this method are also proved in this
paper. To confirm our method feasibly, we construct a detailed 3D air combat
simulation environment for the RL-based methods training firstly, and we
implement our method in both the attack horizontal flight UCAV task and the
self-play confrontation task. Experimental results show that our method
performs better than the methods only utilizing the sparse reward or the
artificial prior experience reward. The agent trained by our method can reach
more than 98.3% win rate in the attack horizontal flight UCAV task and average
67.4% win rate when confronted with the agents trained by the other two
methods.
