Model-aided Deep Reinforcement Learning for Sample-efficient UAV
  Trajectory Design in IoT Networks
  Deep Reinforcement Learning (DRL) is gaining attention as a potential
approach to design trajectories for autonomous unmanned aerial vehicles (UAV)
used as flying access points in the context of cellular or Internet of Things
(IoT) connectivity. DRL solutions offer the advantage of on-the-go learning
hence relying on very little prior contextual information. A corresponding
drawback however lies in the need for many learning episodes which severely
restricts the applicability of such approach in real-world time- and
energy-constrained missions. Here, we propose a model-aided deep Q-learning
approach that, in contrast to previous work, considerably reduces the need for
extensive training data samples, while still achieving the overarching goal of
DRL, i.e to guide a battery-limited UAV on an efficient data harvesting
trajectory, without prior knowledge of wireless channel characteristics and
limited knowledge of wireless node locations. The key idea consists in using a
small subset of nodes as anchors (i.e. with known location) and learning a
model of the propagation environment while implicitly estimating the positions
of regular nodes. Interaction with the model allows us to train a deep
Q-network (DQN) to approximate the optimal UAV control policy. We show that in
comparison with standard DRL approaches, the proposed model-aided approach
requires at least one order of magnitude less training data samples to reach
identical data collection performance, hence offering a first step towards
making DRL a viable solution to the problem.
