Anypath Routing Protocol Design via Q-Learning for Underwater Sensor
  Networks
  As a promising technology in the Internet of Underwater Things, underwater
sensor networks have drawn a widespread attention from both academia and
industry. However, designing a routing protocol for underwater sensor networks
is a great challenge due to high energy consumption and large latency in the
underwater environment. This paper proposes a Q-learning-based
localization-free anypath routing (QLFR) protocol to prolong the lifetime as
well as reduce the end-to-end delay for underwater sensor networks. Aiming at
optimal routing policies, the Q-value is calculated by jointly considering the
residual energy and depth information of sensor nodes throughout the routing
process. More specifically, we define two reward functions (i.e., depth-related
and energy-related rewards) for Q-learning with the objective of reducing
latency and extending network lifetime. In addition, a new holding time
mechanism for packet forwarding is designed according to the priority of
forwarding candidate nodes. Furthermore, a mathematical analysis is presented
to analyze the performance of the proposed routing protocol. Extensive
simulation results demonstrate the superiority performance of the proposed
routing protocol in terms of the end-to-end delay and the network lifetime.
