Federated Double Deep Q-learning for Joint Delay and Energy Minimization
  in IoT networks
  In this paper, we propose a federated deep reinforcement learning framework
to solve a multi-objective optimization problem, where we consider minimizing
the expected long-term task completion delay and energy consumption of IoT
devices. This is done by optimizing offloading decisions, computation resource
allocation, and transmit power allocation. Since the formulated problem is a
mixed-integer non-linear programming (MINLP), we first cast our problem as a
multi-agent distributed deep reinforcement learning (DRL) problem and address
it using double deep Q-network (DDQN), where the actions are offloading
decisions. The immediate cost of each agent is calculated through solving
either the transmit power optimization or local computation resource
optimization, based on the selected offloading decisions (actions). Then, to
enhance the learning speed of IoT devices (agents), we incorporate federated
learning (FDL) at the end of each episode. FDL enhances the scalability of the
proposed DRL framework, creates a context for cooperation between agents, and
minimizes their privacy concerns. Our numerical results demonstrate the
efficacy of our proposed federated DDQN framework in terms of learning speed
compared to federated deep Q network (DQN) and non-federated DDQN algorithms.
In addition, we investigate the impact of batch size, network layers, DDQN
target network update frequency on the learning speed of the FDL.
