{'fraction_non_alphanumeric': 0.025939016354160425, 'fraction_numerical': 0.006499730425927035, 'mean_word_length': 5.021100090171325, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systems

Volt-var control (VVC) is the problem of operating power distribution systems within healthy regimes by controlling actuators in power systems. Existing works have mostly adopted the conventional routine of representing the power systems (a graph with tree topology) as vectors to train deep reinforcement learning (RL) policies. We propose a framework that combines RL with graph neural networks and study the benefits and limitations of graph-based policy in the VVC setting. Our results show that graph-based policies converge to the same rewards asymptotically however at a slower rate when compared to vector representation counterpart. We conduct further analysis on the impact of both observations and actions: on the observation end, we examine the robustness of graph-based policy on two typical data acquisition errors in power systems, namely sensor communication failure and measurement misalignment. On the action end, we show that actuators have various impacts on the system, thus using a graph representation induced by power systems topology may not be the optimal choice. In the end, we conduct a case study to demonstrate that the choice of readout function architecture and graph augmentation can further improve training performance and robustness.


Introduction
Volt-Var Control (VVC) refers to the objective of maintaining the healthy operation of power distribution systems via controlling the voltage (Volt) and reactive power flow (Var). As circuits in power distribution systems usually follow a tree topology, nodes far away from the substation (root node) are usually subject to low voltage magnitude. VVC optimizes the operation of grid assets such as voltage regulators, capacitors, and batteries to minimize power delivery losses as well as to maintain the voltage profile at each bus (node). As such, VVC is one of the critical elements in distribution automation [1] and has been studied for decades [2]. VVC has never been a trivial problem to solve. The operational change of any single grid asset may result in change over the whole power distribution systems, as the nodal voltages/currents are connected by the networked Ohm's law. Researchers have addressed the VVC problem by three different approaches. The first is by sensing the voltage/power profiles near the grid assets and operate the assets based on local observations [3]. This is a simple solution to the VVC problem, which may not necessarily need communication links. Though it is still common practice in the field, the performance of such control could sometimes be poor and requires tremendous effort from the engineers to tune the controllers of the grid assets before/during deployments. 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Sydney, Australia.
With the development of control and communication technology, researchers have formulated the VVC as an optimal power flow (OPF) problem and leveraged optimization solvers to solve it. However, due to the nature of OPF problems, the resulting optimization problem is non-convex and thus hard to solve. Yang et al. [4] is an example that uses linearization techniques, and Gan et al. [5] is another example that uses convex relaxations to circumvent these challenges. These techniques work under certain strong assumptions (operating around the operating point for linearization and voltage around nominal values for convex relaxation). Together with the introduction of integer/discrete control variables modeling actuators, the OPF problem can hardly be scaled to medium and large systems. With recent breakthroughs in RL, power systems researchers have attempted to use RL for power systems operations. One of such examples is learning to operate a transmission systems operation in L2RPN competition [6]. Though transmission systems are fundamentally different from distribution systems in both network topology (looped vs. radial) and typical problem types (dynamic systems vs. quasi-static systems), RL has shown promising results [7] in transmission systems.
In this work, we follow the third approach of utilizing RL to learn an optimal control strategy for the VVC problem and investigate the feasibility of combining such approaches with recent breakthroughs in graph representations to learn better controllers. Specifically, this study attempts to answer the following question: does providing topology information of the power system via a graph representation and using graph convolutional network instead of dense network policies for VVC beneficial to the 1) sample efficiency/convergence rate of training the control policy, 2) robustness of the trained policy to typical data acquisition errors in power systems, and 3) what is a good representation of the underlying circuit topology given the possible different impact of actuators.
This paper contributes to existing literature by being the first attempt to combine graph-based networks with reinforcement learning for VVC. We demonstrate that this graph-based agent and system representation easily scales to power systems of multiple sizes from small (13Bus) to mediumsize systems (8500Nodes), and its performance is competitive with the dense policy counterpart. Additionally, we show that a graph-based policy is much more robust in the presence of missing/noisy observations as compared to a dense-based policy. Furthermore, we show that unlike many other graph representation studies, the actuators in VVC could be using a different topology during message passing compared to the well studied topology induced by physical systems (in our case the circuit). Lastly, we present a case study to investigate the effects of graph augmentation and alternative readout function and show such factors can further improve the performance of the graph-based agent.

Related Works
We briefly go over and highlight some of the recent literature related to this work. With recent advancements in graph neural networks [8], there have been multiple works that leverage the expressive power of graph neural networks with reinforcement learning to solve sequential decisionmaking problems that have inherent graph representations. Some of the applied works include optimizing network routing [9], compilers [10], job scheduling [11] and automatic design of transistor circuit that is transferable [12]. Additionally, the same concept has also been extended applications such as better behavior generation in self-driving applications [13], learning policies that have zeroshot transfer capabilities in robotics [14] and controlling the dynamics of a graph which can be used to represent various real-life scenarios [15].
With reference to power systems domains, RL approaches are a well-studied topic, with RL being applied to a multitude of problems from various aspects [16,17]. In the context of VVC, existing studies were mainly focused on various aspects of scaling RL to the challenges specific to the VVC, such as minimizing constraint violations [18] and scaling to combinatorially large actions spaces [19]. Alternatively, researchers have also tackled the VVC problem by formulating it as multi-agent reinforcement learning (MARL) problem and proposing a novel efficient and resilient MARL algorithm [20]. Additionally, a more recent and closely related work in terms of methodology by [21] also proposed to combine RL with graph neural networks for power system restoration via a multi-agent formulation. Inspired by these related works, we apply the idea of graph-based RL to solve the volt-var control problem using a graph representation as a complementary extension to the works mentioned above.

Volt-var control
VVC can be formulated as an optimal power flow problem, i.e., optimize for an objective function subject to the physical networked constraints. We represent a power distribution system as a tree graph (N , ξ), where N is the set of nodes or buses and ξ is the set of edges or lines and transformers. Let us denote node i as j's parent (radial graph is a tree). In the VVC problem, we consider three different types of actuators, i.e., voltage regulators (reg), capacitors, and batteries. The VVC problem can be described with the following optimization problem [22].
The VVC objective is a combination of three losses: f volt for voltage violation at buses, f ctrl for control error which prevents actuator wear out by penalizing the actuator status from changing too frequently, and f power for power loss. p, q are active and reactive power consumed at buses (nodes) or power flow over lines (edges) , v, denote bus voltage magnitude and squared current magnitude, R, X are resistance and reactance. Using R for real numbers and Z for integers, p bat ∈ R is the battery power output, q cap ∈ Z denotes the capacitor reactive power output. r ∈ Z represents the tapping of voltage regulators. All {r, p bat , q cap } needs to be operating under their operational constraints captured by set S. Capital letters stand for given parameters while lower case letters denotes auxiliary variables, excluding the decision variables. Note that the VVC problem is a time dependent problem, for brevity, we have ignored the time t in all variables. The constraints in Eq. (1) have quadratic equality constraints, making any optimization upon it non-convex.

Reinforcement learning
Based on the previous section, the VVC problem can be formulated as a sequential Markov Decision Problem (MDP) as the power system has no memory and the system's transition into the next state is solely dependent on the control decision and current state. Hence, we can cast the problem as an MDP and solve it via conventional RL approaches, where the objective function is formulated as the reward function in the RL framework. Note that the constraints in the VVC can also be absorbed as part of the reward function or handled explicitly via constrained-RL algorithms [23,24].
Formally, let H denote the horizon of an episode and let f (s t , a t ) denote the underlying environmental dynamics which transitions the system into the next state, s t+1 , according to the current state, s t and action a t based on the physics equation described in the previous section. We parameterized the RL policy, π θ (s t ) using a (graph) neural network, where a t ∼ π θ (s t ). The objective of the RL algorithm is then to optimize parameters of the neural network which maximizes the cumulative reward function, subject to the environment dynamics: Most RL algorithms typically fall between the spectrum of value-based methods and policy optimization methods. Value-based methods such as Q-learning [25] and its variants aim to learn a function that estimates the value of any state-action pairs and the policy proceeds to take action, which maximizes the estimated cumulative reward. On the other hand, vanilla policy optimization methods, i.e., REINFORCE [26] aims to directly learn a reward-maximizing policy by estimating the gradient of the policy and updating the policy. Finally, actor-critic methods such as A2C [27] and SAC [28] borrows heavily from both sides to achieve a much better training performance. In this paper, we focus on the PPO algorithm [29] with the clipped objective and actor-critic architecture as the baseline for comparing different dense versus graph-based policies due to PPO's ability to scale to multi-discrete action spaces.

Graph neural networks
Graph neural networks (GNN) are a special class of neural networks designed to learn from graphstructured data by capturing the relationship between different nodes. Recent literature have illustrated that different GNNs architectures can be designed and applied to various inductive and transductive tasks such as node classification, edge prediction, and graph classification. The core idea of GNNs is to learn embeddings using the message-passing mechanism, where the features of a node in the graph are aggregated based on the features of neighboring nodes, with some learnable parameters transforming the messages. Depending on the downstream application at hand (node classification, graph classification, etc.), the learned nodal embeddings are typically further aggregated and/or sent through a readout function (usually a dense layer) that outputs a final prediction. In this work, we focus specifically on Graph Convolutional Networks (GCN) [30] as a possible representation of π θ . Formally, let h i+1 v denote the features/embeddings of node v at the next layer and let σ denote any arbitrary non-linearity function, b i a learnable bias and b i , φ i the learnable weight term of the current layer. Further, let N (v) denote the neighborhood of node v, E u,v denote the edge weight between nodes (u, v), C u,v some normalization constant and h i u the features/embeddings of the current layer. Then the nodal embeddings of the i + 1 layer can be defined as: Note that the number of layers i is also representative of the depth of the node neighborhood we consider, i.e., the number of hops information is propagated along. In the context of using a GNN as an RL policy, we use the final embeddings h i+1 v as inputs to a readout function, R(.) (i.e., meanpooling of nodal embeddings) to generate the logits to a dense layer which outputs the actions. We further highlight that using other variants of GNNs such as using alternate forms of readout functions, aggregation functions [31] and including attention on edge weights [32] is also possible and is complementary to our approach.

Methodology
To answer the question of whether using a graph representation and GNN-based policy for VVC result in better performances, we trained dense-and graph-based PPO agents on the PowerGym environment [33], which consists of benchmark systems of various sizes. Specifically, we trained separate policies on four power systems: 13Bus, 34Bus, 123Bus, and 8500Node 1 . In these systems, the agent controls different combinations of voltage regulators, capacitors, and batteries to modulate the node voltage across the entire system within a normalized threshold of 0.95 and 1.05V while simultaneously reducing overall control cost and system power loss.

Environment: The reward function of the environments takes the form of
where R v denotes the penalty for voltage violations, R c penalty for changing controls, R p penalty for power loss and α v , α c , α p denotes the respective weightage for each penalty term. These terms map back to the three terms in the objective function of Eq. (1). Note that the terms R v and R c may also be conflicting in practice. Minimizing the voltage violation, R v would require frequent operation of voltage regulators, capacitors, and batteries, which would subsequently increase the control cost, R c and vice versa. Thus, it results in a multi-objective RL scenario which would require careful tuning of α v and α c . Nevertheless, for simplicity, we used the prescribed reward function and weights presented in the benchmark environment. The observations of the PowerGym environment consist of the minimum phase voltage at every bus as well as the status of the controllable actuators, i.e., voltage regulators, capacitors, and batteries. The actions consist of changing the status of the controls, such as changing the tapping of a voltage regulator, switching the capacitors state between open/close, and charging/discharging the batteries. Since the capacitors have only two switchable states while the batteries and voltage regulators have multiple switchable states, the action space is represented by a multi-discrete action space, where each actuator can be controlled with independent and potentially different numbers of discrete actions. For more details on the environment setup, we defer the reader to the article on PowerGym [33].
Representation and training: To compare the training performance of dense versus graph-based policy, we trained the PPO agents on 20,000 episodes, with each episode having a horizon of 24 steps and subjected to different energy load profiles for every episode to introduce variations. For the agent parameterized with the dense policy (Dense-PPO), we concatenated the observation of the environment as a vector input to the policy, as is commonly done in literature. For the agent parameterized by a GCN policy, denoted as Graph-PPO, we constructed a graph representation according to the topology of the physical power systems and assigned every node the features consisting of the node voltage and capacitor, voltage regulator, and battery status. Nodes that do not have capacitors, voltage regulators or batteries are padded with zeros. The entire graph representation of the power system's state is then sent as input to the GCN for Graph-PPO. To facilitate fair comparisons, we used the same number of layers and hidden dimensions for both the dense and GCN network to compare the performance on the same system. All experiments were performed on a server with one AMD Ryzen Threadripper 3970X CPU and one Nvidia RTX 3090 GPU. Further details on training hyper-parameters are provided in the supplementary materials. Figure 1: Rewards achieved by Dense-PPO and Graph-PPO over 20,000 training episodes. Rewards shown are averaged over five random seeds. Graph-PPO performs marginally better than Dense-PPO for smaller systems but converges much slower than Dense-PPO for larger systems.

Training convergence
We first compare the results of training the RL agent using a dense policy versus a GCN-based policy. Figure 1 illustrates the rewards obtain by the RL policy during training for the 13Bus, 34Bus, 123Bus, and 8500Node systems, respectively. There are two keys observations that are worth noting. First, we observe that both Dense-PPO and Graph-PPO are able to converge to similar rewards asymptotically for all four systems. This validates the fact that both policies can learn from their respective representations. The second observation is that for a smaller system (13Bus), Graph-PPO is able to converge to a marginally higher reward than Dense-PPO. Nevertheless, as we increase the size of the system, also corresponding to a larger state representation, we can see that Dense-PPO outperforms Graph-PPO.
From these observations, we conclude that using a graph representation and graph-based policy does not provide any significant benefit in terms of training convergence. In fact, using a graph representation does result in significantly slower training for larger systems (8500Node), which are much closer to the size of power distribution systems in real life. However, this conclusion is subjected to some additional caveats. We hypothesize the reason that Graph-PPO converges slower for larger systems is due to the message passing nature of the graph convolutional layers. Since the number of graph convolutional layers represents the number of hops away message from a node is allowed to propagate to its neighboring nodes, this severely restricts the representation of the system if there is a presence of a node with global effects while the number of graph convolution layers is limited in the GCN. This hypothesis is further validated in subsection 5.3. In contrast, dense layers in Dense-PPO are essentially able to learn correlations between the embeddings of any nodes, regardless of the connectivity of the physical systems, thus allowing Dense-PPO to learn faster. It is worth highlighting that for the experiments shown, we have followed the convention of most state-of-the-art GCNs, which uses only three to four layers deep network, partially due to the small-world properties [34] in such applications. As such, we have used a three-layer GCN for 13Bus, 34Bus, and 123Bus system and a four-layer GCN for the 8500Node system. On the contrary, a traversal of the 123Bus and 8500Node graph reveals a tree depth of 26 and 272, respectively. Furthermore, it has also been shown that using very deep GCNs can lead to the over-smoothing phenomenon [35], where the useful signals in a node's features/embeddings get over-mixed by neighboring nodes.

Robustness to typical data acquisition errors
In power systems, it is not uncommon to have communication failures which results in missing voltage readings (missing observations). Further, voltage reading can be very noisy (noisy observations) due to misalignment of sampling time, i.e., voltage meters measuring different snapshots of the same power system. We examine the benefits and drawbacks of Graph-PPO versus Dense-PPO from a perspective of the robustness of the learned control policy towards the absence of features or presence of noise in the features. While the general robustness of RL policies to adversarial attacks has been widely studied in literature [36,37,38], we investigate the control policy's robustness towards missing and noisy observations/measurements as they are a realistic setting in the context of power distribution systems. To simulate such settings and test the robustness of the Graph-PPO and Dense-PPO, we designed two experiments. In the first experiment, to simulate communication failure (missing observations), we randomly selected a subset of nodes in the power distribution system and set the voltage values to zero during testing. This resulted in certain values in the input vector being zero for Dense-PPO and certain nodal features being zero in the input graph for Graph-PPO. To simulate measurement misalignment (noisy measurements) in the second experiment, we generated uniform random noise, and added them to the voltage values of randomly sampled subsets of nodes.  Table 1: Values in the table denote the percent difference of rewards from the nominal setting with full observations. Each column represents a scenario with a certain percent of missing node voltage observations due to sensor communication failures. Table 1 shows the effects of missing observations for both Dense-PPO and Graph-PPO on all four systems with a different number of node voltage values masked. Each value in the table represents the average percentage decrease in rewards of the agent in the missing observation setting from the nominal setting with full observations available. Each average value was obtained by averaging the drop in performance across five random subsets of nodes and agents trained with different seeds. As observed, though Dense-PPO and Graph-PPO are both negatively affected by missing observations, the effect of missing observations is significantly more severe on Dense-PPO as compared to Graph-PPO. Additionally, we see that the significance of this effect increases as we increase the number of missing observations. In the best-case scenario, Dense-PPO suffers at least 3X worse than Graph-PPO, and in the worst-case scenario, Dense-PPO is 23X worse than Graph-PPO (123Bus with 75% missing observations). Overall, this demonstrates that Graph-PPO is capable of maintaining a much more robust performance than Dense-PPO in the event of missing observations. Additional visualizations illustrating the performance Dense-PPO and Graph-PPO in the nominal and missing observation settings are presented in the supplementary materials.
Measurement misalignment: Table 2 presents the performance of Dense-PPO and Graph-PPO under misaligned measurements. Comparing the scenario of noisy observations to missing observations, we observe that both Dense-PPO and Graph-PPO are less sensitive to noisy observations than missing observations. Nonetheless, we still observe a similar trend where Graph-PPO maintains a much more robust performance in the presence of noisy observations, especially when a larger percentage of the observations are noisy (i.e., 50% and 75% columns in Table 2).
From the results above, we can affirmatively confirm that Graph-PPO enables a much more robust control policy than Dense-PPO in a situation when voltage readings may be missing or noisy. This occurrence can also be attributed to the message-passing mechanism of the GCN in a Graph-PPO. When voltage observations of any nodes are missing or noisy, the GCN architecture enables the information of neighboring nodes to naturally fill in the missing values or smoothen out the noisy values to generate a much more accurate overall state representation. Another possible cause is that the dense layers of Dense-PPO, in the absence of node-connectivity constraints in the vector representations, have learned spurious correlations among non-connected nodes. This has allowed the Dense-PPO to converge much faster during training but also subsequently caused the policy to overfit to the relative positions of voltage observations in the vector representation to a certain degree. In the following sections, we attempt to further investigate this claim via subsection 5.3 and subsection 5.4.

Actuator sensitivity analysis
In the results presented above, we have made an assumption that the graphical representation of the power system follows the topology of the physical power system, i.e., only buses that are connected by lines will have an edge connecting the nodes in the graph. This representation can limit the information propagation between nodes when using a limited number of GCN layers. Thus, changing features of one node will have a larger effect on nodes that are directly connected as compared to nodes that are connected yet far away. However, from the power systems perspective, not all the actuators behave the same way. Certain actuators, such as the voltage regulator, have a global effect, while batteries and capacitors have a more local effect. The following section serves to validate the presence both global and local nodes specific to the domain of power transmission system.

Sensitivity analysis:
We conducted a sensitivity analysis on the power system by running the random policy for a single episode using only one active actuator (while disabling the other actuators) and observed the co-variance of the voltage between the actuator bus and the voltage of all other buses. For easy visualization purposes, we demonstrated this on a 13Bus system, shown in Figure 2. Figure 3 illustrates the co-variances between the buses when only the capacitor, voltage regulator, or battery is active. As observed in Figure 3, we can tell that the voltage regulator clearly has a global effect on every bus, even though not all the buses are directly connected to the voltage regulator bus. In contrast, the capacitor and batteries have a more local effect, with the capacitor having a larger neighborhood influence than the battery.
The property of global versus local effect of physical actuators is a topic that is more oriented towards the domain of engineering controls, and is not present in most of the graph representational learning tasks, which typically assumes the topology used for message passing is induced by the modeling of physical systems. Bearing this observation in mind, we have improved the graph representations in subsection 5.4. Figure 3: Visualization of co-variance between nodes in 13Bus system when only capacitor, voltage regulator or battery is active. Observe that the effects of capacitors and batteries are local while the effect of the voltage regulator is global, although the voltage regulator may not be directly connected to every node.

123Bus Case Study
Based on the findings of the previous section, we have validated that global nodes and local nodes do exist in the power systems. However, from the discussions in subsection 5.1, we also know that the effects of the global nodes aren't properly propagated throughout the entire graph as we are only using three or four graph convolution layers. In this section, we investigate if we can incorporate such knowledge by augmenting the representation of the graph to improve the performance of Graph-PPO. In light of the global and local effects, we also study how different readout functions, R(.) used in the GCN networks can affect the final performance.
Learning with augmented graph representation: To properly represent the existence of a node with global effects such as the voltage regulator, we augmented the graph representation of the power system in the following way: Given the original graph topology based on the physical system, we traversed the graph from the root (source bus) to the leaves and recorded the child nodes of every voltage regulator encountered during the traversal. We then augmented the graph by adding an edge between the nodes with voltage regulator and all its corresponding child nodes. This effectively mimics the global effects of the voltage regulator node by allowing information to propagate in the GCN layers even if a child node is not directly connected to the voltage regulator node.
Learning with a different readout function: We also perform a study on the effects of using a different readout functions. As an initial baseline, we have used a mean-pooling readout function to average the embeddings of all the nodes from the final GCN layer in the policy/value networks before passing them to a dense layer to output the control actions (mean-pool readout). This network design has been used in all the results presented in the previous sections. However, averaging the embeddings of all nodes for decision may not be the best approach, especially when the capacitors and batteries only need to observe the neighborhood voltages to exert local control. We tested an alternative readout function where instead of averaging the embeddings of all nodes, we only take the embeddings of the control actuators and stacked them together to form the logits to the final dense layer (local readout). This architecture corresponds to control actions computed based on the embeddings of the capacitor, voltage regulator, and battery nodes.
Results of graph augmentation and localized readout function: Figure 4 and Table 3 summarizes the results of exploring the graph augmentation and alternative readout function. From Figure 4, we see that when using a mean-pooling readout, augmenting the graph representation provides no effect as the training performances are similar. In contrast, using a local readout function provides a noticeable increase in performance. Finally, using a local readout function in tandem with graph augmentation further improves the performance almost similar to the Dense-PPO. In terms of robustness during testing (Table 3), we observe that the Augmented Graph-PPO with a mean-pooling readout is less robust in terms of communication failure but slightly more robust in terms of measurements misalignment than vanilla Graph-PPO with mean-pooling. However, Augmented Graph-PPO with local readout is consistently the most robust in both settings, while vanilla Graph-PPO using local readout is slightly less robust than Augmented Graph-PPO with local readout. Nevertheless, even the worse performing vanilla Graph-PPO is significantly more robust when compared to using a Dense-PPO, as tabulated in Table 1 and 2. In summary, we have shown that the choice of readout functions can significantly affect the performance of the control policy, and graph augmentation can have added advantages when paired with the proper readout function. Figure 4: Effects of adding using graph augmentation and localized readout function based only on the nodes with actuators. Observe that augmentation had no effect when paired with mean-pooling readout but using local readout and augmentation gives the best improvement  Table 3: Summary of the robustness of the variations of Graph-PPO towards communication failure (missing measurements) and measurement misalignment (noisy measurements). Augmented Graph-PPO with local readout provides the best robustness.

Conclusion and future work
This work presents the first attempt of using a graph representation to train a graph-based deep reinforcement learning policy for volt-var control in power distribution systems. We investigated the benefits and drawbacks of learning an RL-based strategy using a graph representation and graph-based policy (Graph-PPO) instead of the conventional vector representation with a dense policy (Dense-PPO). Our results reveal that Graph-PPO can achieve the same rewards as Dense-PPO asymptotically. However, Dense-PPO does converge faster than Graph-PPO for larger systems (i.e., larger graphs). The second result of this work is that while Graph-PPO is slower to train, using a graph representation along with the graph-based policy provides a much more robust control in the event of sensor communication failure and measurement misalignment. In addition, our case study demonstrates that augmentation of the graph representation to account for the presence of nodes with global effects and using an alternative readout function based only on local node embeddings can further improve Graph-PPO's training performance and robustness. Future directions of this work include incorporating advanced graph neural networks architectures as part of the policy representation, extending this framework to more complex settings such as larger and heterogeneous graphs (power systems coupled with communication networks), and investigating the transferability of these graph-based controllers to different power systems topologies.