A Q-Learning-based Approach for Distributed Beam Scheduling in mmWave
  Networks
  We consider the problem of distributed downlink beam scheduling and power
allocation for millimeter-Wave (mmWave) cellular networks where multiple base
stations (BSs) belonging to different service operators share the same
unlicensed spectrum with no central coordination or cooperation among them. Our
goal is to design efficient distributed beam scheduling and power allocation
algorithms such that the network-level payoff, defined as the weighted sum of
the total throughput and a power penalization term, can be maximized. To this
end, we propose a distributed scheduling approach to power allocation and
adaptation for efficient interference management over the shared spectrum by
modeling each BS as an independent Q-learning agent. As a baseline, we compare
the proposed approach to the state-of-the-art non-cooperative game-based
approach which was previously developed for the same problem. We conduct
extensive experiments under various scenarios to verify the effect of multiple
factors on the performance of both approaches. Experiment results show that the
proposed approach adapts well to different interference situations by learning
from experience and can achieve higher payoff than the game-based approach. The
proposed approach can also be integrated into our previously developed Lyapunov
stochastic optimization framework for the purpose of network utility
maximization with optimality guarantee. As a result, the weights in the payoff
function can be automatically and optimally determined by the virtual queue
values from the sub-problems derived from the Lyapunov optimization framework.
