Optimal Beam Association for High Mobility mmWave Vehicular Networks:
  Lightweight Parallel Reinforcement Learning Approach
  In intelligent transportation systems (ITS), vehicles are expected to feature
with advanced applications and services which demand ultra-high data rates and
low-latency communications. For that, the millimeter wave (mmWave)
communication has been emerging as a very promising solution. However,
incorporating the mmWave into ITS is particularly challenging due to the high
mobility of vehicles and the inherent sensitivity of mmWave beams to dynamic
blockages. This article addresses these problems by developing an optimal beam
association framework for mmWave vehicular networks under high mobility.
Specifically, we use the semi-Markov decision process to capture the dynamics
and uncertainty of the environment. The Q-learning algorithm is then often used
to find the optimal policy. However, Q-learning is notorious for its
slow-convergence. Instead of adopting deep reinforcement learning structures
(like most works in the literature), we leverage the fact that there are
usually multiple vehicles on the road to speed up the learning process. To that
end, we develop a lightweight yet very effective parallel Q-learning algorithm
to quickly obtain the optimal policy by simultaneously learning from various
vehicles. Extensive simulations demonstrate that our proposed solution can
increase the data rate by 47% and reduce the disconnection probability by 29%
compared to other solutions.
