{'fraction_non_alphanumeric': 0.047802236945216216, 'fraction_numerical': 0.011968369309681556, 'mean_word_length': 4.260445943413903, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reinforcement Learning for Distributed Transient Frequency Control with Stability and Safety Guarantees

This paper proposes a reinforcement learning-based approach for optimal transient frequency control in power systems with stability and safety guarantees. Building on Lyapunov stability theory and safety-critical control, we derive sufficient conditions on the distributed controller design that ensure the stability and transient frequency safety of the closed-loop system. Our idea of distributed dynamic budget assignment makes these conditions less conservative than those in recent literature, so that they can impose less stringent restrictions on the search space of control policies. We construct neural network controllers that parameterize such control policies and use reinforcement learning to train an optimal one. Simulations on the IEEE 39-bus network illustrate the guaranteed stability and safety properties of the controller along with its significantly improved optimality.


I. INTRODUCTION
With modern power systems shifting from high-inertia traditional generations to low-inertia renewable resources, it is increasingly important to design control mechanisms that allow to operate frequency around its nominal value. To tackle the frequency control problem, the appeal of learning methods lies in the convenience of incorporating large amounts of data and accounting for optimality considerations in the controller design. This paper serves as a contribution to the growing body of work that seeks to leverage learning in the synthesis of efficient decision-making mechanisms in power systems that have rigorous guarantees on stability and performance.
Literature Review. Transient stability of power systems refers to its ability to regain operating equilibrium after disturbances, while retaining the state within operational margins. The literature has investigated optimal frequency control design for improving transient stability, e.g., load-side control [1], [2], droop coefficient design [3], and proportionalderivative control [4], to mention a few. These methods either rely on designing optimal linear feedback controllers offline or solving optimization problems in real time to obtain optimal control policies. While these approaches ensure transient stability, they do not strictly guarantee transient safety, as the frequency may enter unsafe regions before convergence. To address this, [5] combined Lyapunov stability analysis and safety-control methods to ensure both stability and transient safety. This approach was further combined with model predictive control in [6], [7] to minimize the control effort by enhancing the cooperation among the nodes, at the cost of a significantly heavier computational burden.
Recent research has employed data-driven methods to improve frequency control design without the restriction for the controllers to be linear or the need to solve computationally complex optimization problems in real time. Reinforcement learning (RL) has emerged as an attractive method to learn such control policies offline, see e.g., [8]. In general, stability and safety of the closed-loop system are not guaranteed without additional design constraints on the learned policies. This has resulted in a number of works that developed stability [9]- [13] or safety [14]- [16] guaranteed RL approaches to learn optimal controllers for frequency [9], [10], [13], [14] and voltage control [11], [12], [15], [16] in power systems. With respect to previous work, the main novelty here is that we develop an RL-based approach that jointly guarantees stability and transient frequency safety. To achieve this, we go beyond purely decentralized controller designs and leverage the distributed cooperation among agents, so that they can share the disturbance to be balanced and ensure system stability.
Statement of Contributions. We study optimal transient frequency control in power systems with dynamics described by the swing equations. We formulate an optimization problem to identify control designs that minimize the frequency deviation from the equilibrium and the control cost over time while ensuring asymptotic stability and transient frequency safety in the presence of disturbances. Leveraging notions of Lyapunov stability and safety-critical control, we identify constraints on the distributed controller design whose satisfaction automatically guarantees that the closed-loop system remains stable and the transient frequency stays within the desired safety bounds. These constraints use budgets to break down the requirement of collectively satisfying an inequality to ensure stability into individual stability conditions, one per bus, in a way that is distributed and allows additional design flexibility for certain buses while having others compensate for it. These constraints define the search space of distributed, stable, and safe control policies. We leverage them to enforce appropriate structural constraints on neural networks so that the resulting parameterized controller belongs to the search space and can approximate with arbitrary accuracy any of its elements. Finally, we use a recurrent neural network (RNN)based RL framework to learn the optimal parameters for these neural networks. Simulation results of the designed controllers on the IEEE 39-bus power system validate their guaranteed stability and transient safety as well as their significantly improved optimality compared to previous controller designs.

II. PRELIMINARIES
We introduce here some notations and basic notions from the algebraic graph theory and the swing dynamics for power systems.

A. Notations
Throughout this paper, we use N, R, R ≥0 and R >0 to denote the set of natural, real, nonnegative and positive real numbers, arXiv:2207.03329v1 [eess.SY] 7 Jul 2022 respectively. For a, b ∈ N, let [a, b] N {x ∈ N | a x b}. For C ⊂ R n , ∂C denotes its boundary. For A ∈ R m×n , [A] i and [A] ij represent its i-th row and (i, j)-th element, respectively. We denote by A † its unique pseudo-inverse and by Range(A) its column space. 1 n and 0 n in R n are vectors of all ones and zeros, respectively. A continuous function α : R → R is of (extended) class-K if it is strictly increasing and α(0) = 0. Finally, · 1 , · and · ∞ are respectively 1-norm, Euclidean norm and infinity norm.

B. Graph theory
Here we present some basic notions in graph theory [17]. Let G = (I, E) be an undirected graph, where I = {1, . . . , n} is the node set and E = {e 1 , . . . , e m } ⊆ I × I is the edge set. Two nodes are neighbors if there exists an edge linking them. We denote by N i the set of neighbors of node i. A path is an ordered sequence of nodes such that any pair of consecutive nodes in the sequence is an edge of the graph. The graph G is connected if there exists a path between any pair of nodes. The adjacency matrix A is defined by [A] ij > 0 if i and j are neighbors, 0 otherwise. The Laplacian matrix L is defined as The value 0 is an eigenvalue of L with eigenvector 1 n . This eigenvalue is simple if and only if the graph is connected. For each edge e k ∈ E with nodes i, j, we assign an arbitrary orientation so that either i or j is the source of e k and the other node is the target of e k . Then the incidence matrix B = (d ik ) ∈ R n×m of graph G is defined as if node i is the source of edge e k −1 if node i is the target of edge e k 0 otherwise.

C. Power network dynamics
The power network is modeled by a connected undirected graph G = (I, E), where I = {1, . . . , n} is the set of buses and E ⊆ I × I is the set of transmission lines. We assume each bus represents an aggregate area consisting of loads and generators. For each bus i ∈ I, we use θ i ∈ R, ω i ∈ R, p i ∈ R, u i ∈ R to represent its voltage angle, frequency deviation (from the nominal value), uncontrolled active power injection, and controlled active power injection, respectively. The frequency dynamics is described by the swing equations [18]:θ are the inertia and damping coefficients of bus i, respectively, and b ij ∈ R >0 is the susceptance of the transmission line connecting buses i and j. For simplicity, we assume M i , D i , b ij are all positive. Define vectors θ [θ 1 , . . . , θ n ] ∈ R n , ω [ω 1 , . . . , ω n ] ∈ R n and p [p 1 , . . . , p n ] ∈ R n . Let B ∈ R n×m be the incidence matrix under an arbitrary graph orientation, and define the voltage angle difference vector Denote by Y b ∈ R m×m the diagonal matrix with [Y b ] k,k = b ij , for each edge k = 1, 2, . . . , m that links nodes i and j, and define M diag(M 1 , M 2 , . . . , M n ) ∈ R n×n , D diag(D 1 , D 2 , . . . , D n ) ∈ R n×n . We rewrite the dynamics (1) in a compact form in terms of λ(t) and ω(t) aṡ where u(t) [u 1 (t), . . . , u n (t)] ∈ R n , and sin λ(t) ∈ R m is the component-wise sine value of λ(t). Note that the transformation (2) enforces λ(0) ∈ Range(B ). We refer to any λ(0) satisfying this condition as an admissible initial value. For convenience, we use x(t) (λ(t), ω(t)) ∈ R m+n to denote the collection of all the state variables, and skip writing its dependence on t if the context is clear. Let According to [19], if where y E,∞ max (i,j)∈E |y i − y j |, then there exists λ ∞ ∈ Γ {λ | |λ k | < π/2, ∀k = 1, ..., m} unique in Γ cl {λ | |λ k | ≤ π/2, ∀k = 1, ..., m} such that , which is asymptotically stable.

III. PROBLEM FORMULATION
Consider a power network modeled as in Section II. Under the condition (4), the unforced system admits a unique locally asymptotically stable equilibrium point (λ ∞ , ω ∞ 1 n ). However, in the presence of disturbances, the transient frequency can enter unsafe regions before convergence to the equilibrium. This can be caused, for instance, by a sudden change in load or generation. If such frequency excursions exceed certain bounds, they might in fact lead to failure of loads or generators. To address this issue, we need to design feedback controllers {u i } i∈I to ensure each nodal frequency ω i stays within its safety bounds [ω i , ω i ] during transients, while preserving the asymptotic stability of (3). We also seek to minimize the frequency deviation from the equilibrium and the control cost integrated over time. This gives rise to where T is the time horizon of interest and γ is a coefficient balancing control cost and frequency deviation. Here p = p nom + ∆p, where p nom is the nominal power injection and ∆p accounts for the disturbance. We assume ∆p vanishes in finite time, and hence only affects the transient behavior. The term γ ω − ω ∞ 1 n 2 in the objective function penalizes deviation from the equilibrium frequency, and can be interpreted as a soft constraint to provide approximate transient safety. Instead, the safety bound in (5d) is a hard constraint to strictly guarantee transient safety by prohibiting the frequency nadir going outside the safe region. We also require the designed controllers to be distributed, in the sense that each bus can implement u i (x, p) using its local information and the information from its neighboring buses and incident transmission lines.
The infinite-dimensional and nonlinear nature of the optimization (5) makes it hard to solve. Reinforcement learning (RL) is an attractive approach to such a problem by employing the data from system executions to train a policy that maps states to input actions. This results in a learned controller with optimized performance for the given data, but does not guarantee the stability and safety of the closed-loop system. Instead, model-based methods leverage knowledge of the dynamics to synthesize feedback controllers that render the system stable and safe, but have trouble dealing with the infinite-dimensional nature of the optimization. The advantages and limitations of RL and model-based approaches motivate us to combine them by identifying conditions on the controller design that ensure stability and safety (cf. Section IV) and incorporating these conditions in the RL policy search (cf. Section V).

IV. SEARCH SPACE OF CONTROL POLICIES
Here we identify constraints on the control design that ensure transient frequency safety and asymptotic stability. These constraints define later the search space of control policies.

A. Constraint ensuring frequency invariance
We first turn our attention to the identification of conditions on the controller design that ensure the transient safety requirement, i.e., ω i (t) staying in [ω i , ω i ] for all i ∈ I and all To make this set forward invariant, one simply needs to ensure that the time-derivative of the frequency is negative when ω i = ω i , positive when ω i = ω i , and anything when ω i ∈ (ω i , ω i ). However, such specification may result in discontinuous controllers. Instead, we seek a specification that gradually kicks in as the frequency reaches certain thresholds, while retaining the stability properties of (3) in the absence of input when the frequency is inside the thresholds. Meanwhile, a dead zone [ω th i , ω th i ] is introduced to avoid over-reaction of the controller to small frequency deviations. Built on this idea, the next result identifies a sufficient condition for a continuous controller design to ensure forward invariance of the frequency-safe set Q i .
Lemma IV.1. (Sufficient condition for frequency invariance [5,Lemma 4.4]): Assume the solution of (3) exists and is unique for every admissible initial condition. For each i ∈ I, let ω th i , ω th i ∈ R be such that ω i < ω th i < ω th i < ω i and α i (·) and α i (·) be class-K functions. If for all x ∈ R m+n and p ∈ R n , where B. Constraint ensuring asymptotic stability Here we derive a constraint on the control design that ensures asymptotic stability. We approach this by considering an energy function and restricting the input so that its timederivative along the closed-loop dynamics is nonpositive. Following [5], [20], consider To ensureV (λ, ω) ≤ 0, one can simply ask u i (x, p) to satisfy for each i ∈ I. This stability condition is convenient, from a network perspective, because it provides an individually decoupled constraint for each bus. This is essentially the approach taken in our previous work [5] and also in [9]. Nevertheless, one can see that it is over-constraining, as the sum of all terms in (8) is what needs to be nonpositive, not each individual summand. One could envision scenarios where some buses can deal with larger disturbances than others. In such cases, it would be advantageous to allow less capable buses to violate (9) up to a level that can be compensated by more capable buses to still make the overall sum (8) with β ∈ R >0 . Suppose for every i ∈ I, x ∈ R m+n , and p ∈ R n , Then, provided λ(0) ∈ Range(B ) and (λ(0), ω(0)) ∈ J β for some β > 1, 1) The solution of the closed-loop system exists and is unique for all t ≥ 0; 2) λ(t) ∈ Range(B ) and (λ(t), ω(t)) ∈ J β for all t ≥ 0; 3) (λ ∞ , ω ∞ 1 n ) is stable, and lim t→∞ (λ(t), ω(t)) = (λ ∞ , ω ∞ 1 n ).
Proof. Note that J β is non-empty and compact. Hence if 2) holds, then 1) follows [21, Theorems 3.1 and 3.3]. Therefore we focus on the statements 2)-3). From (8), Hence, given (λ(0), ω(0)) ∈ J β , we have V (λ, ω) ≤ V (λ(0), ω(0)) ≤ c/β, and 2) follows. For 3), note that it is easy to see from (3)  The quantities {b i } i∈I ∈ R n in Lemma IV.2 correspond to the budgets that allow some buses to violate the local condition (9) and instead satisfy while ensuring system stability as long as Note that the condition (11) is more general than the stability condition in [5,Lemma 4.1], which requires u i to have a different sign from (ω i − ω ∞ ), and the stability condition in [9, Theorem 1], which further requires u i to be monotonically decreasing with ω i − ω ∞ .

C. Distributed dynamic budget assignment
Before proceeding to the synthesis of distributed control policy, here we focus on the assignment of budgets introduced previously. Notice that both sufficient conditions (6) and (11) obtained above are naturally distributed, except for the requirement that n i=1 b i = 0. Indeed, the satisfaction of this equality requires coordination across the buses. Interestingly, a static, a priori budget assignment in general does not work. This is because, if b i = 0, then (11) might require the control input to be infinitely large (instead of vanishing) when ω i approaches ω ∞ . Instead, we need a dynamic, state-dependent budget assignment that makes sure b i approaches zero as ω i approaches ω ∞ . The following result details a dynamic budget assignment mechanism that ensures this while guaranteeing n i=1 b i = 0 in a distributed way. Proposition IV.3. (Distributed dynamic budget assignment): For x ∈ R m+n , let I th ⊆ I denote the set of buses satisfying ω i / ∈ [ω th i , ω th i ] and E th ⊆ E the set of edges between any pair of nodes in I th . Define the (possibly unconnected) statedependent subgraphG = (I, E th ) of G = (I, E) and letL be its Laplacian matrix. Given ξ ∈ R n , let the budgets be assigned as b i = [L] i ξ for each i ∈ I. Then the following holds: 1) The underlying idea of Proposition IV.3 is that, instead of assigning b i directly to each bus i ∈ I, we let each bus i ∈ I choose a value ξ i by itself and compute b i by exchanging information with its neighbors, utilizing the algebraic properties of Laplacian matrices to enforce n i=1 b i = 0. For bus i ∈ I, whenever ω i (t) ∈ [ω th i , ω th i ], the mechanism does not include it in the budget assignment process, and in that case its budget b i is simply zero. Finally, we remark that other dynamic budget assignment mechanisms, different from the one proposed here, might also work.

D. Distributed, stable and safe control policies
Here, we combine the results of the previous sections to identify the search space of distributed policies. In the next result, we propose a policy design to satisfy (6) and (11), which renders the closed-loop system stable and safe.
Theorem IV.4. (Distributed control policies with asymptotic stability and transient safety guarantees): Given thresholds Under condition (4), consider the system (3) with a Lipschitz control policy satisfying (12) with budgets b i = [L] i ξ for each i ∈ I, whereL is defined in Proposition IV.3. If λ(0) ∈ Range(B ) and (λ(0), ω(0)) ∈ J β for some β > 1, then the following holds: 1) The solution of the closed-loop system exists and is unique for all t ≥ 0; 2) λ(t) ∈ Range(B ) and (λ(t), ω(t)) ∈ J β for all t ≥ 0; for all t > 0; 5) The budgets vanish in finite time, i.e., there exists a time t 0 > 0 such that b i = 0 for all t > t 0 and all i ∈ I.
Proof. Statements 1)-4) readily follow Lemmas IV.1 and IV.2 if (12) (i) ensures that (6) and (11) hold and (ii) defines a specification that can be satisfied by a Lipschitz controller. For (i), from Proposition IV.3 we have n i=1 b i = 0, and therefore (12) implies both (6) and (11). For (ii), notice that the only problem is that (12) requires u i = 0 when ω i ∈ [ω th i , ω th i ]. Hence, to guarantee it admits a Lipschitz controller, we need to show that u i can be chosen as 0 right after ω i passes the thresholds ω th i and ω th i . Hence, it suffices to show that lim ωi→(ω th Now we only show the case when ω i → (ω th i ) + , since the other case can be proved similarly. Note that lim ωi→(ω th ωi−ω th i + q i (x, p) = +∞, and hence the minimum is attained by the first term. We end the proof by noting that lim ωi→(ω th Finally, for 5), the asymptotic convergence established in 3) indicates that there exists a t 0 > 0 such that ω th i ≤ ω i (t) ≤ ω th i holds for all t > t 0 and all i ∈ I. Together with Proposition IV.3, the result follows.
Theorem IV.4 provides a characterization of the search space of distributed control policies that guarantee asymptotically stable and transient-safe closed-loop systems. Fig. 1 illustrates the search space. compensates it up to a certain amount to ensure the overall system stability.

V. SYNTHESIS OF DISTRIBUTED NEURAL NETWORK CONTROLLERS
We construct neural networks that parameterize control policies satisfying the requirements in Section IV-D, and then apply an RNN-based RL framework to train an optimal one.

A. Selecting class-K functions and frequency thresholds
The condition (12) obtained in Theorem IV.4 depends upon the class-K functions α i , α i and the frequency thresholds ω th i , ω th i . Their choice affects the search space of control policies. One can make specific choices for these design parameters according to practical considerations. Alternatively, one can use neural networks to parameterize and train them along with the control policy. Note that parameterizing ω th i and ω th i is easy since they are static values instead of functions. The more difficult task of parameterizing α i and α i requires the neural networks to be strictly monotone. Approaches to this include structure-based [22] and verification-based [23] methods. Here, we adopt the single hidden layer monotone neural network design in [9], which achieves universal approximation, yet is easy to implement. We provide next the details of the proposed parameterization method.
Then, α i is of class-K. Furthermore, for any class-K function κ and given any compact domain K ⊂ R and We omit the proof of this result, but note that it is analogous to the proof of [9,Theorem 2]. The underlying idea is to construct a piece-wise linear approximation of a nonlinear function in which every linear segment is strictly increasing. An arbitrary accuracy of the approximation can be achieved given sufficiently many neurons. Also note that α i can be constructed in the same way with weight vectors z + i , z Then ω th i and ω th i approximate any values in (ω i , ω ∞ ), and (ω ∞ , ω i ), respectively.
The proof of Lemma V.2 readily follows the definition of the sigmoid function.

B. Neural network controller design
We first give the final ingredient to parameterize control policies that satisfy condition (12) using neural networks. The next result provides a parameterization of any function Moreover, for any Lipschitz function g i : R → R satisfying g i (ω i ) = 0 for ω i ∈ [ω th i , ω th i ] and given any compact domain K ⊂ R and > 0, there exists q The proof of this result uses the definition of ReLU function and exploits a piece-wise linear approximation similar to that in Lemma V.1.
The proof readily follows the universal approximation results in Lemmas V.1, V.2 and V.3.

C. Learning optimal control policy using RNN
Having parameterized in Theorem V.4 the search space identified in Section IV-D, here we describe an approach to train an optimal control policy adopting the RNN-based RL framework proposed in [9]. To simulate the trajectories for training the neural network controller (13), we use a firstorder Euler discretization with stepsize ∆t for problem (5). Let K be the total number of timesteps. The discrete-time optimization problem is where u φ = [u 1,φ , . . . , u n,φ ] . The learning algorithm works as follows. At the beginning of the training process, all parameters in φ are randomly generated. Training is implemented in a batch updating style, where the initial states ω(0) and λ(0) in each batch are randomly generated. In each episode, we use the current control policy u φ to generate state trajectories of length K for all batches through dynamics (14b),(14c), and compute the loss function (14a). The trainable parameters φ are updated by gradient descent on the loss function (14a) and converge to a local optimum.
Remark V.5. (Robust policy learning through Lipschitz regularization): One way to enhance the robustness of the learned control policy is to add an additional regularization term in the cost function to promote the control policy by a small Lipschitz constant [24]. Here we implement this Lipschitz-regularized learning by adding an additional term ρ 1 where the parameter ρ controls the trade-off between robustness and optimality of the learned control policy. We implement this reguralization in the simulations below and illustrate its added robustness against state measurement noise. •

VI. CASE STUDY
We conduct a case study to illustrate the performance of the proposed approach. We consider the IEEE 39-bus power network and assume each bus represents an aggregate area containing loads and generators. The system parameters are from [5], [25]. We consider the time horizon of interest to be T = 50 seconds and bus 38 encountering a sudden change in power injection during the time interval (0, 2] seconds, with ∆p 38 = −p nom 38 . The nominal frequency is 60 Hz, and the safe region is set as [59.8, 60.2] Hz for every bus. Simulation Setup. We build the RL environment using TensorFlow 2.7.0 and conduct the training process in Google Colab on a single TPU with 32 GB memory. We set the discretization stepsize ∆t at 0.0008 seconds. To facilitate the training process, we only evaluate the first 10 seconds in each episode, meaning the total number of stages K in each episode is 12500. For each i ∈ I, ω i (0) is randomly generated in [59.9, 60.1] Hz, and λ(0) is calculated using power injections randomly generated over [0.9p nom i (0), 1.1p nom i (0)]. The balancing coefficient in the objective function is γ = 40, and the number of episodes, the batch size, and the number of neurons m are 150, 50, 20, respectively. We use Adam algorithm [26] to update parameter φ in each episode with learning rate 0.05.
Baseline for Comparison. We conduct two comparisons for illustration. The first is to compare our approach to the methods proposed in [9] and [5]. The controllers in [9] are parameterized as non-increasing functions passing through the origin and then trained via RL, and the controllers in [5] are designed as u i (x, p) = min{0, αi(ωi−ωi) Simulation Results. Fig. 2 illustrates the performance of the proposed RL-based method, the RL-based method in [9], and the method in [5] with the same randomly generated initial states. Table I summarizes the comparison results. The approach proposed here guarantees both asymptotic stability and transient safety, while significantly reducing the cost. Fig. 3 shows the dynamic budget allocation under the proposed assignment mechanism, validating that the budget for each bus vanishes when its frequency converges while keeping the summation of all the budgets equal to zero. In this way, more nodes can contribute to the transient frequency regulation while cooperatively minimizing the cost. The proposed method also achieves faster convergence and smaller transient fluctuations. Fig. 4 compares the performance of the proposed method with and without Lipschitz regularization in the presence of frequency measurement noise. In both cases the frequency nadir slightly violates the safety bound due to the noisy input to controllers. However, the Lipschitz regularization helps enhance system robustness by reducing the fluctuations in state and control. VII. CONCLUSIONS We have presented a reinforcement learning approach to the synthesis of optimal controllers that are distributed and guarantee the stability and transient safety of power networks.  Leveraging notions of Lyapunov stability and safety-critical control, we have identified conditions on the controller design that ensure stability and transient frequency safety. These constraints incorporate the idea of endowing some buses with additional design flexibility through budgets in a way that collectively ensures the stability of the overall system. We have constructed neural networks to parameterize the control policies within the identified search space and employed an RL framework to learn an optimal controller. Simulations illustrate the guaranteed stability and transient frequency safety of the resulting closed-loop system while showing a significant reduction in the cost. Our future work aims to further improve the dynamic budget allocation to reduce cost and transient fluctuation, while incorporating higher-order dynamics of generators, inverter-interfaced energy resources, and their existing control loops.