PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in
  Power Systems
  We present the PowerGridworld software package to provide users with a
lightweight, modular, and customizable framework for creating
power-systems-focused, multi-agent Gym environments that readily integrate with
existing training frameworks for reinforcement learning (RL). Although many
frameworks exist for training multi-agent RL (MARL) policies, none can rapidly
prototype and develop the environments themselves, especially in the context of
heterogeneous (composite, multi-device) power systems where power flow
solutions are required to define grid-level variables and costs. PowerGridworld
is an open-source software package that helps to fill this gap. To highlight
PowerGridworld's key features, we present two case studies and demonstrate
learning MARL policies using both OpenAI's multi-agent deep deterministic
policy gradient (MADDPG) and RLLib's proximal policy optimization (PPO)
algorithms. In both cases, at least some subset of agents incorporates elements
of the power flow solution at each time step as part of their reward (negative
cost) structures.
