{'fraction_non_alphanumeric': 0.04474015968623057, 'fraction_numerical': 0.009497128449362655, 'mean_word_length': 4.501849568434032, 'pattern_counts': {'":': 2, '<': 4, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A game-theoretic approach to dynamic demand response management

Within the realm of dynamic of \emph{smart buildings} and \emph{smart cities}, dynamic response management is playing an ever-increasing role thus attracting the attention of scientists from different disciplines. Dynamic demand response management involves a set of operations aiming at decentralizing the control of loads in large and complex power networks. Each single appliance if fully responsive and readjusts its energy demand to the overall network load. A main issue is related to mains frequency oscillations resulting from an unbalance between supply and demand. In a nutshell, this paper contributes to the topic by equipping each signal consumer with strategic insight. In particular, we highlight three main contributions and a few other minor contributions. First, we design a mean-field game for the TCLs application, study the mean-field equilibrium for the deterministic mean-field game and investigate on asymptotic stability for the microscopic dynamics. Second, we extend the analysis and design to imperfect models which involve both stochastic or deterministic disturbances. This leads to robust mean-field equilibrium strategies guaranteeing stochastic and worst-case stability, respectively. Minor contributions involve the use of stochastic control strategies rather than deterministic, and some numerical studies illustrating the efficacy of the proposed strategies.


Introduction
Demand response management involves a set of operations aiming at decentralizing load control in power networks [3,13,14,31]. In particular, it calls for the alteration of the timing, of the level of instantaneous demand, or of the total electricity by end-use customers from their normal consumption patterns in response to changes in the price of electricity over time. This is possible also through an opportune design of incentive payments to induce lower electricity use at off-peak times.
A communication protocol aggregates relevant information on the past, current and forecasted demand and transmits it to each fully responsive load controller or decision mechanism, which will adopt opportune actions such as increasing or decreasing the proper load or energy demand. The novelty of this paper is in that fully responsive load control together with the many cooperative and competitive aspects involved in the process, are now reviewed as a game with a large number of indistinguishable players, these being the single loads. For illustrative purposes, in this paper, fully responsive load control is reviewed in the context of thermostatically controlled loads (TCLs), in smart buildings or plug-in electric vehicles [4,22,23,25,29], see Fig. 1. A first idea of this work, which is common also to [4,6], is to adopt stochastic response strategies rather than deterministic. This means that each TCL selects a probability with which to activate one of the two functioning modes, on and off. Thus a probability value of 1/2 means that the TCL is 50% on and 50% off. It has been shown in [4,6] that stochastic response strategies outperform deterministic ones, especially in terms of attenuating the mains frequency oscillations. These are due to the unbalance between energy demand and supply (see e.g. [26]). The mains frequency usually needs to be stabilized around a nominal value (50 Hz in Europe). If electrical demand exceeds generation then frequency will decline, and vice versa.
A qualitative plot of such an oscillatory phenomenon is displayed in Fig. 2. The two rows depict the time plot of the state of each TCL, namely the temperature in the top row and the mode of functioning in the bottom row. Here each TCL increases or decreases its proper load in response to the current network load, and as clear visually, this induces oscillations in the mains frequency due to an undesired synchronized reaction of the whole population of TCLs.
This preamble introduces the main aim of this paper, which studies constructive design methods of distributed demand response management strategies in order to reduce the mains frequency oscillations and stabilize both the temperature and the functioning mode of the TCLs. The model used in this paper is as follows. Each single TCL is a player and is characterized by two state variables, the temperature and the functioning mode. The state dynamics of a TCL -henceforth referred to as microscopic dynamics to distinguish it from the dynamics of the aggregate temperature and functioning mode of the whole population, the latter called macroscopic dynamics -describes the time evolution of its temperature and mode in the form of a linear ordinary differential equation in the deterministic case, and of a stochastic differential equation in the stochastic case. In addition to the state dynamics, each TCL is programmed with a given finite-horizon cost functional that accounts for i) energy consumption, ii) deviation of mains frequency from the nominal one, and iii) deviation of the TCL's temperature from a reference value. More formally, the mains frequency involved in the specifics ii) is used in a cross-coupling meanfield term that incentivizes the TCL to switch to of f if the mains frequency is below the nominal value and to switch to on if the mains frequency is above the nominal value. In other words, the cross-coupling mean-field term models all kinds of incentive payments, benefits, or smart pricing policies aiming at shifting demand from high-peak to off-peak periods.

Highlights of contributions
This paper provides three main results. First, in the spirit of prescriptive game theory and mechanism design [5] we design a mean-field game for the TCLs application, study the mean-field equilibrium for the deterministic mean-field game and investigate on asymptotic stability for the microscopic dynamics. Asymptotic stability means that both the temperature and the mode functioning of each TCL converges to the reference value. A second result relates to the stochastic case, characterized by a stochastic disturbance in the form of a Brownian motion in the microscopic dynamics. After establishing a meanfield equilibrium, we provide some results on stochastic stability. In particular, we focus on two distinct scenarios. In one case, we assume that the stochastic disturbance expires in a neighborhood of the origin. This reflects in having the Brownian motion coefficients linear in the state. The resulting dynamics is well-known in the literature as geometric Brownian motion. As for any geometric Brownian motion, we can study conditions for it to be stochastically stable almost surely. This means that the state trajectories converge to zero with probability one. In a second case, the stochastic disturbance is independent on the state and the Brownian motion coefficients are constant. This leads to a dynamics which resembles the Langevin equation. Following well-known results on the Langevin equation, the dynamics is proven to be stochastically stable in the second-moment. An expository work on stochastic analysis and stability is [21]. A third result deals with ro-bustness for the microscopic dynamics. The dynamics is now influenced by an additional adversarial disturbance, with bounded resource or energy. Even for this case, we study the mean-field equilibrium and investigate on conditions that guarantee worst-case stability.

Literature overview
We introduce next two streams of literature. One is related to dynamic response management, while the second one is about the theory of differential games with a large number of indistinguishable players, also known as mean-field games.

Related literature on demand response
Examples of papers developing the idea of dynamic demand management are [11,12,22,23]. In particular, [11] provides an overview on the redistribution of the load away from peak hours and the design of decentralized strategies to produce a predefined load trajectory. This idea is further developed in [12]. To understand the role of game theory in respect to this specific context the reader is referred to [22]. There, the authors present a large population game where the agents are plug-in electric vehicles and the Nashequilibrium strategies (see [8]) correspond to distributed charging policies that redistribute the load away from peaks. The resulting strategies are known with the name of valleyfilling strategies. In this paper we adopt the same perspective in that we show that network frequency stabilization can be achieved by giving incentives to the agents to adjust their strategies in order to converge to a mean field equilibrium. To do this, in the spirit of prescriptive game theory [5], a central planner or game designer has to design the individual objective function so to penalize those agents that are in on state in peak hours, as well as those who are in of f state in off-peak hours. Valley-filling and coordination strategies have been shown particularly efficient in thermostatically controlled loads such as refrigerators, air conditioners and electric water heaters [23].
The results obtained in this paper are in accordance with the recent results in [4], according to which, stochastic control laws are in general more appropriate than deterministic ones when it comes to desynchronize the appliances functioning.

Related literature on mean-field games
A second stream of literature related to the problem at hand is on mean-field games. Mean-field games were formulated by Lasry and Lions in [20] and independently by M.Y.
Huang, P. E. Caines and R. Malhamé in [18,19]. The mean-field theory of dynamical games is a modeling framework at the interface of differential game theory, mathematical physics, and H ∞ -optimal control that tries to capture the mutual influence between a crowd and its individuals.
From a mathematical point of view the mean-field approach leads to a system of two PDEs. The first PDE is the Hamilton-Jacobi-Bellman (HJB) equation. The second PDE is the Fokker-Planck-Kolmogorov (FPK) equation which describes the density of the players. Explicit solutions in terms of mean-field equilibria are available for linearquadratic mean-field games [7], and have been recently extended to more general cases in [15]. In addition to explicit solutions, a variety of solution schemes have been recently proposed based on discretization and/or numerical approximations, see e.g. [1,2,24]. The idea of extending the state space, which originates in optimal control [27,28], has been also used to approximate mean-field equilibria in [9]. More recently, robustness and risk-sensitivity have been brought into the picture of mean-field games [10,30], where the first PDE is now the Hamilton-Jacobi-Isaacs (HJI) equation. For a survey on mean-field games and applications we refers the reader to [16]. A first attempt to apply mean-field games to demand management is in [6].
The paper is organized as follows. In Section 2 we state the problem and introduce the model. In Section 3 we review some preliminary results. In Section 4 we state and discuss the main results. In Section 6 we provide some discussion. In Section 5 we carry out some numerical studies. Finally, in Section 7 we provide some conclusions.

Notation
The symbol E indicates the expectation operator. We use ∂ x and ∂ 2 xx to denote the first and second partial derivatives with respect to x, respectively. Given a vector x ∈ R n and a matrix a ∈ R n×n we denote by x 2 a the weighted two-norm x T ax. The symbol a i• means the ith row of a given matrix a. We denote by Diag(x) the diagonal matrix in R n×n whose entries in the main diagonal are the components of x. We denote by dist(X, X * ) the distance between two points X and X * in R n . We denote by Π M (X) the projection of X onto set M. The symbol ":" denotes the Frobenius product.

Population of TCLs through mean-field games
In this section, in the spirit of prescriptive game theory and mechanism design [5], we design a mean-field game for the TCLs application, with the aim of incentivizing cooperation among the TCLs through an opportune design of distributed cost functionals, one per each TCL.
Consider a population of hybrid controlled thermostat loads (TCLs) and a time horizon . Each TCL is characterized by a continuous state, namely the temperature x(t), and a binary state π on (t) ∈ {0, 1}, representing the condition on or of f at time t ∈ [0, T ]. When the TCL is set to on the temperature decreases exponentially up to a fixed lower temperature x on whereas in the of f position the temperature increases exponentially up to a higher temperature x of f . Then, the temperature of each appliance evolves according to the following differential equations: with initial state x(0) = x and where the rates α, β are given positive scalars.
In accordance with [4,6] we set the problem in a stochastic framework where each TCL is in one of the two states on or of f with given probabilities π on ∈ [0, 1] and π of f ∈ [0, 1]. The control variable is the transitioning rate u on from of f to on and the transitioning rate u of f from on to of f . This is illustrated in the automata in Fig. 3. The corresponding dynamics is then given by Asπ on (t) +π of f (t) = 0, we can simply consider only one of the above dynamics. Then, let us denote y(t) = π on (t) and introduce a stochastic disturbance in the form of a Brownian motion, denote it B(t), and a deterministic disturbance For any x, y in the "set of feasible states" S :=]x on , x of f [×]0, 1[, the resulting dynamics in a very general form is given by where σ ij and d ij , i, j = 1, 2 are positive scalar coefficients. For a mean-field game formulation, consider a probability density function m : which satisfies At every time t the network or mains frequency depends linearly on the discrepancy between the percentage of TCLs in on position and a nominal value. We call such a discrepancy as error and denote it by e(t) = m on (t) − m on , where m on is the nominal value (the higher the percentage of TCLs in on position with respect to the nominal value, the lower the network frequency).
We then consider the running cost below, which depends on the distribution m(x, y, t) through the error e(t): where q, r on , r of f , and S are opportune positive scalars.
Note that cost (4) includes four terms. The term 1 2 qx(t) 2 penalizes the deviation of the TCLs' temperature from the nominal value, which we set to zero. The terms 1 2 r on u on (t) 2 introduces a cost for fast switching; namely this cost is zero when either u on (t) = 0 (no switching) and is maximal when u on (t) = 1 (probability 1 of switching). Similar comment applies 1 2 r of f u of f (t) 2 . The term y(t)Se(t) accounts for the network stabilization in that it penalizes those appliances that are on whenever e(t) > 0, the latter condition meaning that demand exceeds supply. The same term turns into a revenue if an appliance is on whenever e(t) < 0, i.e., whenever supply exceeds demand. Finally, the penalty term y(t)W accounts for the minimization of power, namely, whenever the TCL is on the power consumption is W .
Also consider a terminal cost Ψ : to be yet designed.
Problem statement. Given a finite horizon T > 0 and an initial distribution m 0 : minimize over U and maximize over W , subject to the controlled system (3), the cost functional

Preliminary results
This section reviews first-and second-order mean-field games in preparation to apply the game to the problem at hand. In the first case, the microscopic dynamics is deterministic and the resulting mean-field game involves only the first derivatives of the value function and of the density function. In the second case, the microscopic dynamics is a stochastic differential equation driven by a Brownian motion, which leads to the involvement of second derivatives of the value function and density function. In addition to this, this section specializes the model to the case under study, involving a population of TCLs and introduced in the previous section.

First-and second-order mean-field games
This section streamlines some preliminary results on mean-field games. To this purpose, consider a generic cost and dynamics T t=0 c(X(t), m, U(.))dt + Ψ(X(T )), where c(.) is the running cost, Ψ(X) ∀ X ∈ in R n is the terminal penalty, and where U(.) is any state-feedback closed loop control policy. Let v(X, t) be the value function, i.e., the optimal value of J(X, t, U(·)). Then from [20] it is well-known that the problem results in the following mean-field game system The partial differential equation (PDE) 6 (a) is the Hamilton-Jacobi-Bellman equation which returns the value function v(X, t) once we fix the distribution m(X, t); This PDE has to be solved backwards with boundary conditions at final time T , represented by the last line in 6 (a). In 6 (b) we have the optimal closed-loop control U * (X, t) as maximizer of the Hamiltonian function in the rhs. The PDE 7 represents the transport equation of the measure m immersed in a vector field F (X, U * (X)); It returns the distribution m(X, t) once fixed the the optimal closed-loop control U * (X, t) and consequently the vector field F (X, U * (X)). Such a PDE has to be solved forwards with boundary condition at the initial time (see the last line) of (7). In a second order mean-field game, the dynamics is a stochastic differential equation driven by a Brownian motion, and the cost function is considered through its expected value, namely, where B(t) ∈ R n is the Brownian motion and σ(X) ∈ R n×n is the coefficient matrix.
From [20] the second-order mean-field game system is then given by where the symbol ":" denotes the Frobenius product andσ ij = n k=1 σ ik (X)σ jk (X). In a second-order mean-field game the Hamilton-Jacobi-Bellman equation, as in 9 (a), involves the second-order derivatives of the value function in the additional term represented by the Frobenius product; Likewise, also the transport equation as in (10) involves the second-order derivatives of the density function. The rest of the system is similar to the first-order case. Let us now specialize the above model to the TCLs application introduced in the previous section.

Mean-field game for the TCL application
Specializing to our TCLs application, let v(x, y, m, t) be the value function, i.e., the optimal value of J(x, y, t, u(·)). Let us denote by Then, the problem at hand can be rewritten as in terms of the state, control and disturbance vectors and yields the linear quadratic problem: where and D = d 11 d 12 The resulting mean-field game is given by and whereσ ij = n k=1 σ ik (X)σ jk (X). Essentially, the partial differential equation (PDE) (14) (a) is the Hamilton-Jacobi-Isaacs equation which returns the value function v(x, y, m, t) once we fix the distribution m(x, y, t); This PDE has to be solved backwards with boundary conditions at final time T , represented by the last line in 14 (a). In 14 (b) we have the optimal closed-loop control u * (x, t) and worst-case disturbance w * (x, t) as min-maximizers of the Hamiltonian function in the RHS. The PDE (15) represents the transport equation of the measure m immersed in a vector field AX + Bu + C + Dw; It returns the distribution m(x, y, t) once fixed both u * (x, t) and w * (x, t) and consequently the vector field AX + Bu * + C + Dw * . Such a PDE has to be solved forwards with boundary condition at the initial time (see the fourth line) of 15. Finally, once given m(x, y, t) from (c) and entered into the running cost c(x, y, m, u) in (a), we obtain the error Note thatX 1] xm(x, y, t)dxdy 1] ym(x, y, t)dxdy , and therefore, henceforth we can refer to as mean-field equilibrium solutions any pair (v(X, t),X(t)) which is solution of (14)-(15).

Main results
This paper contributes in three directions with respect to the TCLs application introduced earlier. First, it analyzes and computes the mean-field equilibrium for the deterministic mean-field game and proves that under certain conditions the microscopic dynamics is asymptotically stable. We repeat the analysis for the stochastic case, assuming that the microscopic dynamics is uncertain. Even for this case, a mean-field equilibrium is computed, and stochastic stability is studied. We distinguish two cases. On the one hand, we consider a stochastic disturbance which fades to zero the closer the state is to zero. The Brownian motion coefficients are linear in the state and the resulting dynamics is also known as geometric Brownian motion. On the other hand, we take the stochastic disturbance being independent on the state. The Brownian motion coefficients are constant and the resulting dynamics mirrors the Langevin equation. In both cases we prove stochastic stability of second-moment for the stochastic process at hand. This section ends with a detailed analysis of robustness properties. The microscopic dynamics is now subject to an addition exogenous input, the disturbance, with bounded resource or energy. Even for this case, we study the mean-field equilibrium and investigate on condition that guarantee stability.

Mean-field equilibrium and stability
In this section we establish an explicit solution in terms of mean-field equilibrium for the deterministic case and study stability of the microscopic dynamics. This case is obtained by fixing to zero the coefficients of both stochastic and adversarial disturbance.
The linear quadratic problem we wish to solve is then: The next result shows that the problem reduces to solving three matrix equations.
Theorem 1 (Mean-field equilibrium) Let D, Σ = 0 in the game (14)- (15). A mean-field equilibrium for (14)- (15) is given by where Furthermore, the mean-field equilibrium strategies are given by Proof. Given in the appendix. Let us note that by substituting the mean-field equilibrium strategies u * = −R −1 B T [P X+ Ψ] given in (20) in the open-loop microscopic dynamicsẊ(t) = AX(t) + Bu(t) + C as defined in (17), the closed-loop microscopic dynamics iṡ Now, let X be the set of equilibrium points for (21), namely, the set of X such that and let V (X(t)) = dist(X(t), X ). The next result establishes a condition under which the above dynamics converges asymptotically to the set of equilibrium points.
Proof. Given in the appendix.

Stochastic case
In this section we study the case where the dynamics is given by a stochastic differential equation driven by a Brownian motion. In other words, the model is uncertain and the uncertainty is modeled as a stochastic disturbance.
The problem at hand is then: where all matrices are as in (12) and This section investigates on the solution of the HJI equation under the assumption that the time evolution of the common state is given. We show that the problem reduces to solving three matrix equations. To see this, by isolating the HJI part of (14) for fixed m t , for t ∈ [0, T ], we have Let us consider the following value function v(X, t) = 1 2 X T P (t)X + Ψ(t) T X + χ(t), so that (48) can be rewritten as The boundary conditions are obtained by imposing that v(X, T ) = 1 2 X T P (T )X + Ψ(T )X + χ(T ) = 1 2 X T φX.

Case I: state dependent variance
The first case we consider involves coefficients for the Brownian motion linear in the state, namely Theorem 2 (stochastic mean-field equilibrium: case I) A mean-field equilibrium for the game (9) with Σ(X) as in (27) is given by where Furthermore, the mean-field equilibrium strategy is Proof. Given in the appendix. Based on the above result, let us now substitute the expression of the mean-field equilibrium strategy u * = −R −1 B T [P X + Ψ] as in (31) in the open-loop microscopic dynamics dX(t) = (AX(t) + Bu(t) + C)dt + ΣdB(t) given in (23) so to obtain the closed-loop microscopic dynamics Now, let X be the set of equilibrium points for (32), namely, the set of X such that and let V (X(t)) = dist(X(t), X ). The next result establishes a condition under which the above dynamics converges asymptotically to the set of equilibrium points.
Corollary 2 (2nd moment boundedness) Let a compact set M ⊂ R 2 be given. Suppose that for all X ∈ M then dynamics (32) is a stochastic process with 2nd moment bounded.
Proof. Given in the appendix.

Case II: state independent variance and Langevin equation
The second case we consider involves coefficients for the Brownian motion which are constant, namely Theorem 3 (stochastic mean-field equilibrium: case II) Let Σ be as in (34). A mean-field equilibrium for the game (14)-(15) is given by (37) Furthermore, the mean-field equilibrium strategies are given by Proof. Given in the appendix. Based on the above result, let us now substitute the expression of the mean-field equilibrium strategy u * = −R −1 B T [P X + Ψ] as in (38) in the open-loop microscopic dynamics dX(t) = (AX(t) + Bu(t) + C)dt + ΣdB(t) given in (23) so to obtain the closed-loop microscopic dynamics Now, let X be the set of equilibrium points for (21), namely, the set of X such that and let V (X(t)) = dist(X(t), X ). The next result establishes a condition under which the above dynamics converges asymptotically to the set of equilibrium points.
Corollary 3 (2nd moment boundedness) Let a compact set M ⊂ R 2 be given. Suppose that for all X ∈ M then dynamics (39) is a stochastic process with 2nd moment bounded.
Proof. Given in the appendix.

Model miss-specification
This section deals with model miss-specification, this being represented by an additional exogenous and adversarial disturbance. The disturbance is supposed to be of bounded energy. Thus, the linear quadratic problem we wish to solve is: This section investigates on the solution of the HJI equation under the assumption that the time evolution of the common state is given. We show that the problem reduces to solving three matrix equations. To see this, by isolating the HJI part of (14) for fixed m t , for t ∈ [0, T ], we have Theorem 4 (worst-case mean-field equilibrium) A mean-field equilibrium for (14)-(15) is given by where Furthermore, the mean-field equilibrium control and disturbance are Now, let X be the set of equilibrium points for (21), namely, the set of X such that and let V (X(t)) = dist(X(t), X ). The next result establishes a condition under which the above dynamics converges asymptotically to the set of equilibrium points.
Proof. Given in the appendix.

Numerical studies
In this section a system consisting of n = 10 2 indistinguishable TCLs. All simulations are carried out with MATLAB on an Intel(R) Core(TM)2 Duo, CPU P8400 at 2.27 GHz and a 3GB of RAM. The number of iterations is T = 30. We consider a discrete time version of (17) The parameter are as shown in Table 1 and in particular the step size dt = 0.1, the cooling and heating rates are α = β = 1, the lowest and highest temperatures are x on = −10, and x of f = 10, respectively, the penalty coefficients are r on = r of f = 1, and q = 1, and the initial distribution is normal with zero mean and standard deviation std(m(0)) =

1.
Input: Set of parameters as in Table 1.
The optimal control is taken as where P is obtained from running the MATLAB command [P]=care(A,B,Q,R), which receives the matrices as input and returns the solution P to the algebraic Riccati equation. Under the assumption BR −1 B T Ψ ≈ C the resulting closed-loop dynamics is given by In contrast with what we observed in Fig. 2, the TCLs show a stable behavior. The simulation is carried out assuming that any 10 seconds the states are subject to an impulse. The TCLs react to the impulse very fast and converge to the equilibrium point before a new impulse is activated, as clear visually in the plot.
We repeat the simulation for the two stochastic cases discussed earlier. The stochastic version of the dynamics appears now as X(t + dt) = X(t) + (A(x(t))X(t) + Bu(t) + C + ΣW (t))dt or for the first case, and X(t + dt) = X(t) + (A(x(t))X(t) + Bu(t) + C + Σ(x)W (t))dt for the second case. Here W (t) is a random walk. The corresponding closed-loop dynamics are then respectively. Figure 5  The experiment is repeated in Figure 6 for the geometric Brownian motion. As in the previous cases the figure displays the time plot of the state of each TCL, namely its temperature x(t) (top row) and mode y(t) (bottom row) in the first case. As the Brownian motion is not weighted by the state (in modulus), its effects are attenuated and the plot is more similar to the one in Fig. 4.
Note that except for the Langevin-type dynamics, in the remainder two cases the TCLs states are driven to zero. For the Langevin-type dynamics the state is confined within a neighborhood of zero.

Discussion
With regards to the problem at hand, the topic of dynamic response management has sparked the attention of scientists from different disciplines. This is witnessed by the rapid growing of publications in journals of different research areas, from differential game theory [6,12,25], to control and optimization [4,11,22,23,26], to computer science [29]. One reason for this is that dynamic response management intersects research programs in smart buildings and smart cities. The problem is relevant due to an ever-increasing size of network systems and the consequent impossibility of centralizing the management of the whole system.
Fully aware of the importance of the topic, let us discuss the relevance of the results of this paper. First, it must be said that the game-theoretic approach presented here is a natural way to deal with larges scale, complex and distributed systems where no central planner may be capable of processing all information data and in order to control the whole system online. One way to deal with this issue, and which is the main idea of dynamic demand management, aims at assigning part of the regulation burden to the consumers by using frequency responsive appliances. In other words, each appliance regulates automatically and in a decentralized fashion its power demand based on the mains frequency.
In this respect, the provided model builds upon the strategic interaction among the electrical appliances. Note that here we look at the problem in more general terms and talk about electrical appliances rather than TCLs. The model suits the case where where the latter are numerous and indistinguishable. Indistinguishable means that any appliance in the same condition will react at the same way. We wish to highlight that indistinguishability is not a limitation, as in the case of heterogeneity of the electrical appliances, more complex multi-population models may be derived based on the same modeling approach used here. The results provided in this paper shed light on the existence of mean-field equilibrium solutions. By this we mean strategies based on the current and forecasted demand, which are proven to attenuate oscillations of mains frequency. A first feature of the model at hand is that the considered strategies are stochastic. This means that the TCL sets a probability with which to switch on or of f . Stochastic linear strategies are designed as closed-loop feedback strategies on current state, temperature and switching mode.
Such strategies are computed over a finite horizon and therefore are based on forecasted demand. From another angle, we may say that mean-field equilibrium strategies represent the asymptotic limit of Nash equilibrium strategies, and as such they are the best-response strategies of even single player, for fixed behavior of other players. The proven stability of the microscopic dynamics confirms the asymptotic convergence of the TCLs's states to an equilibrium point, this being expressed in terms of temperature and switching mode. The several cases studied in the paper have shown that this holds true in the cases of both perfect and imperfect modeling. This is a clear evidence of a certain degree of robustness characterizing the proposed strategies. In the case of imperfect modeling, model mis-specifications is considered both in a stochastic and deterministic worst-case scenario. Assuming imperfect models both with a stochastic or worst-case deterministic disturbance acting on the state dynamics, conditions for convergence of the microscopic dynamics are provided.

Concluding remarks
We have illustrated robust mean-field games as a paradigm for crowd-averse systems. We have discussed these systems in the context of stock market, production engineering, and dynamic demand management in power systems. As main contributions we first have formulated the problem as a robust mean-field game; second, we have presented a new approximation method based on the extension of the state space; third we have discussed a relaxation method to minimize the approximation error. Further results are obtained for a scalar microscopic dynamics, for which we have established performance bounds, and analyzed stochastic stability of both the microscopic and the macroscopic dynamics. We can extend our study in at least three directions. These include i) the extension of the approximation method to more general cost functionals, ii) the study of the case with "local" mean-field interactions rather than "global" as in the current scenario, and iii) the analysis of crowd-seeking scenarios in contrast to the crowd-averse cases analyzed in this paper.

Appendix
Proof of Theorem 1 Let us start by isolating the HJI part of (14). For fixed m t and for t ∈ [0, T ], we have which in a more compact form can be rewritten as Let us consider the following value function v(X, t) = 1 2 X T P (t)X + Ψ(t) T X + χ(t), and the corresponding optimal closed-loop state feedback strategy Then (48) can be rewritten as The boundary conditions are obtained by imposing that Since (50) is an identity in x, it reduces to three equations: To understand the influence of the congestion term on the value function, let us now develop the expression for Ψ and obtain The expression of Ψ then can be rewritten as which is of the form     Ψ 1 + aΨ 1 + bΨ 2 + c = 0, From the above set of inequalities, we obtain the solution Ψ(x(t), e(t), t). Note that the term a ′ depends on x and c ′ depends on e(t).
Substituting the expression of the mean-field equilibrium strategies u * = −R −1 B T [P X+ Ψ] as in (20) in the open-loop microscopic dynamicsẊ(t) = AX(t) + Bu(t) + C introduced in (17), and averaging both LHS and RHS we obtain the following closed-loop macroscopic dynamicṡX 1] Ψ(x, e, t)m(x, y, t)dxdy and this concludes our proof.

Proof of Corollary 1
Let X(t) be a solution of dynamics (21) Taking the limit of the difference above we obtaiṅ which implies LV (X(t)) < 0, for all X(t) ∈ X and this concludes our proof.
Reviewing (26) as an identity in x, this leads to the following three equations to solve in the variable P (t), Ψ(t), and χ(t): (58)

Proof of Corollary 2
Let X(t) be a solution of dynamics (32) with initial value (59) From the definition of infinitesimal generator LV (X(t)) = lim dt→0 From (33) the above implies that LV (X(t)) < 0, for all X(t) ∈ M and this concludes our proof.
Again, since the HJB equation (26) is an identity in x, it reduces to three equations: (62) Substituting the expression of the mean-field equilibrium strategy u * = −R −1 B T [P X + Ψ] as in (38) in the open-loop microscopic dynamics dX(t) = (AX(t)+Bu(t)+C)dt+ΣdB t given in (23) and averaging both LHS and RHS we obtain the following closed-loop macroscopic dynamicṡ X(t) = [A − BR −1 B T P ]X(t) − BR −1 B T Ψ * (X(t)) + C, and this concludes our proof.
From (40) the above implies that LV (X(t)) < 0, for all X(t) ∈ M and this concludes our proof.
The above set of identities in x yields the following three equations in the variable P (t), Ψ(t), and χ(t):