Age of Information Aware VNF Scheduling in Industrial IoT Using Deep
  Reinforcement Learning
  In delay-sensitive industrial internet of things (IIoT) applications, the age
of information (AoI) is employed to characterize the freshness of information.
Meanwhile, the emerging network function virtualization provides flexibility
and agility for service providers to deliver a given network service using a
sequence of virtual network functions (VNFs). However, suitable VNF placement
and scheduling in these schemes is NP-hard and finding a globally optimal
solution by traditional approaches is complex. Recently, deep reinforcement
learning (DRL) has appeared as a viable way to solve such problems. In this
paper, we first utilize single agent low-complex compound action actor-critic
RL to cover both discrete and continuous actions and jointly minimize VNF cost
and AoI in terms of network resources under end-to end Quality of Service
constraints. To surmount the single-agent capacity limitation for learning, we
then extend our solution to a multi-agent DRL scheme in which agents
collaborate with each other. Simulation results demonstrate that single-agent
schemes significantly outperform the greedy algorithm in terms of average
network cost and AoI. Moreover, multi-agent solution decreases the average cost
by dividing the tasks between the agents. However, it needs more iterations to
be learned due to the requirement on the agents collaboration.
