Integrating LEO Satellites and Multi-UAV Reinforcement Learning for
  Hybrid FSO/RF Non-Terrestrial Networks
  A mega-constellation of low-altitude earth orbit (LEO) satellites (SATs) and
burgeoning unmanned aerial vehicles (UAVs) are promising enablers for
high-speed and long-distance communications in beyond fifth-generation (5G)
systems. Integrating SATs and UAVs within a non-terrestrial network (NTN), in
this article we investigate the problem of forwarding packets between two
faraway ground terminals through SAT and UAV relays using either
millimeter-wave (mmWave) radio-frequency (RF) or free-space optical (FSO) link.
Towards maximizing the communication efficiency, the real-time associations
with orbiting SATs and the moving trajectories of UAVs should be optimized with
suitable FSO/RF links, which is challenging due to the time-varying network
topology and a huge number of possible control actions. To overcome the
difficulty, we lift this problem to multi-agent deep reinforcement learning
(MARL) with a novel action dimensionality reduction technique. Simulation
results corroborate that our proposed SAT-UAV integrated scheme achieves 1.99x
higher end-to-end sum throughput compared to a benchmark scheme with fixed
ground relays. While improving the throughput, our proposed scheme also aims to
reduce the UAV control energy, yielding 2.25x higher energy efficiency than a
baseline method only maximizing the throughput. Lastly, thanks to utilizing
hybrid FSO/RF links, the proposed scheme achieves up to 62.56x higher peak
throughput and 21.09x higher worst-case throughput than the cases utilizing
either RF or FSO links, highlighting the importance of co-designing SAT-UAV
associations, UAV trajectories, and hybrid FSO/RF links in beyond-5G NTNs.
