{'fraction_non_alphanumeric': 0.03160049592019145, 'fraction_numerical': 0.005593518438428049, 'mean_word_length': 4.605947955390334, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Demand-Side Scheduling Based on Deep Actor-Critic Learning for Smart Grids

We consider the problem of demand-side energy management, where each household is equipped with a smart meter that is able to schedule home appliances online. The goal is to minimise the overall cost under a real-time pricing scheme. While previous works have introduced centralised approaches, we formulate the smart grid environment as a Markov game, where each household is a decentralised agent, and the grid operator produces a price signal that adapts to the energy demand. The main challenge addressed in our approach is partial observability and perceived non-stationarity of the environment from the viewpoint of each agent. We propose a multi-agent extension of a deep actor-critic algorithm that shows success in learning in this environment. This algorithm learns a centralised critic that coordinates training of all agents. Our approach thus uses centralised learning but decentralised execution. Simulation results show that our online deep reinforcement learning method can reduce both the peak-to-average ratio of total energy consumed and the cost of electricity for all households based purely on instantaneous observations and a price signal.


I. INTRODUCTION
In the past decade, Demand-Side Management (DSM) with price-based demand response has been widely considered to be an efficient method of incentivising users to collectively achieve better grid welfare [1]. In a typical DSM scenario, an independent utility operator plans the preferred demand volume for a certain metering time duration within the considered micro (e.g., district) grid. In order to achieve this, the operator adopts a time-varying pricing scheme and announces the prices to the users on a rolling basis. In general, the price may be determined by factors such as the demand for the utility, the supply of the utility, and the history of their volatility. The adjusted price is released in advance at some time window to the consumers. In response to the updated electricity prices, the users voluntarily schedule or shift their appliance loads with the aim of minimising their individual costs, hence making it possible to reduce the energy waste or load fluctuation of the considered microgrid (see the example in [2]).
In the framework of dynamic pricing (a.k.a. real-time pricing), dynamic demand response on the user side (e.g., at residential households) relies on the premise of the availability of smart meters and automatic energy-consumption-manager modules installed in each household. This is mainly due to two characteristics of residential utility demand. Firstly, as Joash Lee is on a scholarship awarded by the Energy Research Institue @ NTU (ERI@N). Wenbo Wang and Dusit Niyato are with the School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore 639798 noted in [1], residential consumers are usually risk-averse and shorted-sighted; it would be very difficult for consumers to schedule for long-term cost optimisation based solely on a dynamic price signal. Secondly, compared with the commercial or industry consumers, individual residential consumers often appear to exhibit near-random consumption patterns [3]. Real-time sensing and communication of data between each home's energy-consumption manager and the utility operator thus becomes essential.
Under the framework of automatic price-demand scheduling, a variety of dynamic-response problem formulations have been proposed in the literature. One major category of the existing schemes focuses on direct load control to optimise a certain objective with (possibly) the constraints on grid capacity and appliance requirement. Typical objectives in this scenario can be to minimise energy consumption/costs [4]- [9], maximise the utility of either middlemen or end-users [10]- [14], or minimise the peak load demand of the overall system [2], [3], [9]. With such a setup, DSM is usually assumed to be performed in a slotted manner (e.g., with a time scale measured in hours [2]) within finite stages. The DSM problem can be formulated under a centralised framework as a scheduling problem [3], or an optimisation problem such as a linear program [9] or a stochastic programs [10]. Alternatively, under a decentralised framework, the DSM problem can also be developed in the form of noncooperative games with specifically designed game structure to guarantee desired social welfare indices [2]. Apart from the programming/planning oriented schemes, another major category of design emphasises the uncertainties in the DSM process, e.g., due to the random arrival of appliance tasks or the unpredictability of the future energy prices. In such a scenario, DSM planning based on the premise of known system model is typically replaced by DSM strategy-learning schemes, which usually address the uncertainty in the system by formulating the considered dynamics to be Markov Decision Processes (MDPs) with partially observability [15]. By assuming infinite plays in a stationary environment, optimal DSM strategies are typically learned in various approaches of reinforcement learning without knowing the exact system models [4], [6].
In this paper, we propose a DSM problem formulation with dynamic prices in a microgrid of residential district, where a number of household users are powered by a single energy aggregator. By equipping the households with energy managerenabled smart meters, we aim at distributedly minimising the cost of energy in the grid, under real-time pricing, over the long term with deep reinforcement learning. The main advantage of decentrally controlled energy-managing smart meters is that, after the training phase, each household agent is able to act independently in the execution phase, without requiring specific information about other households -each household agent will not be able to observe the number or nature of devices operated by other households, the amount of energy they individually consume, and when they consume this energy. Our aforementioned aim addresses at least two key challenges. Firstly, the model of the microgrid system and its pricing methodology is not available to each household. Secondly, the limitation in the observation available to each household and the lack of visibility regarding the strategy of other households creates the local impression of a nonstationary environment. We introduce details of the system model in Section III.
To overcome the first mentioned key challenge, we propose a model-free policy-based reinforcement learning method in Section IV for each household to learn their optimal DSM strategies. To address the second key challenge, we propose a centralised critic network which allows the aggregator to provide an allowable level of messaging to guide the learning processes of the households. Our experiments, presented in Section V, show that the proposed model is able to not only address the scenarios of cooperative learning with shared utility, but also able to adapt themselves with non-cooperative goals of load demands. By comparing to an offline method that plans consumption based on a predicted price schedule [9], we show that our method is better able to handle the uncertainties associated with real-time pricing.

II. RELATED WORK AND PRELIMINARIES
A number of previous works have applied reinforcement learning to train decision-making entities in smart grid settings [7], [11]- [13], [16]- [18]. Of these, [7], [12], [13], [18] have applied Q-learning, a canonical value-based reinforcement learning algorithm, to optimise energy costs. While they vary in the exact method of defining the state and action spaces in the MDPs, a common theme was the presence of electricity producers, consumers, and service providers that brokered trading between them. The reward function in these studies were designed to minimise the cost to the service provider and/or consumer, minimise the discrepancy between the supply and demand of energy or, in the case of [14], maximise the profit achieved by the service provider. A common limitation between [7], [12], [13], [18] was the use of tabular Q-learning limited the algorithm to processing only discretised state and action spaces.
Recently, there have also been studies that have introduced methods utilising deep neural networks to multi-agent reinforcement learning, albeit to different classes of problems [19]- [21]. These newly introduced methods are extensions of the now-canonical deep Q-network (DQN) algorithm for the multi-agent environment, and a common technique is to have a central value-function approximator to share information. However, as explained in section III, a value-based RL algorithm would not suit the problem introduced in this paper because it requires the Markov assumption that the current state and action are sufficient to describe both the state transition model and the reward function.
In this paper, we propose a policy-based algorithm based on the Proximal Policy Optimisation (PPO) algorithm [22] for a multi-agent framework. We consider the partially observable Markov game framework [23], which is an extension of the Markov Decision Process (MDP) for multi-agent systems. A Markov game for N agents can be mathematically described by a tuple N , S, {A n } n∈N , {O n } n∈N , T , {r n } n∈N , where N is the set of individual agents and s S describes the overall state of all agents in the environment.
At each discrete time step k, each agent n is able to make an observation O n of its own state. Based on this observation, each agent takes an action that is chosen using a stochastic policy π θn : O n → A n , where A n is the set of all possible actions for agent n. The environment then determines each agent's reward as a function of the state and each agent's action, such that r n (k) : S × A ∞ × · · · × A n → . It moves on to the next state according to its state transition model T : S × A ∞ × · · · × A n → S .

III. SYSTEM MODEL
We consider a DSM problem with a set of N households that each have M household appliances. Each household is equipped with a smart meter that is capable of communicating and controlling all household appliances. The problem that we present is a more advanced form of the demand-side scheduling problem than that presented in [9] in that each household has no advanced knowledge of upcoming tasks. We primarily consider interactive background appliances. These appliances are actively turned on by the household, but the actual time of operation poses a less immediate impact to the user. The appliances that we consider in each household can be, for example, a washing machine, a clothes dryer, a storage water heater, a dish washer and a refrigerator.
Each household's local operational state can be expressed as s n = [x n , t n , l n , q n ] , where t n = [t 1 n , . . . , t M n ] denotes the vector of the length of time before the next task for appliance m can commence, l n = [l 1 n , . . . , l M n ] denotes the vector of task periods, and q n = [q 1 n , . . . , q M n ] denotes the number of tasks queued for each appliance. The private observation of each household consists of its local state augmented with the price of electricity at the previous time step p(k −1), such that o n = [x n , t n , l n , q n , p(k − 1)] . The overall state is simply the joint operational state of all the users: The task arrival for each appliance corresponds to the household's demand to turn it on, and is based on a four-week period of actual household data from the Smart* dataset [24]. We first discretise each day into a set of H time intervals H = {0, 1, 2, ..., H − 1} of length T , where each interval corresponds to the wall-clock time in the simulated system. The wall-clock time interval for each time-step in the Markov game is determined with the following relation: Next, we count the number of events in each interval where the above-mentioned appliances of interest were turned on. This is used to compute an estimate for the probability each appliance receives a new task during each time interval, p m n (h(k)). The task arrival for each appliance q m n at each time interval k is thus modelled with a Bernoulli distribution. Each task arrival q m n for a particular appliance m in household n is added to a queue q m n . The duration of the next task to be executed is denoted by l m n . Its value is updated by randomly sampling from an exponential distribution whenever a new task is generated (i.e. q m n = 1). The rate parameter λ m n for the distribution for each appliance is assigned to be the reciprocal of the approximate average duration of tasks in the above-mentioned four-week period selected from the Smart* dataset [24]. We choose to keep the task duration constrained to l ∈ [T, ∞].
The use of the described probabilistic models for generation of task arrivals and task lengths enable us to capture the variation in household energy demand throughout the course of an average day while preserving stochasticity in energy demand at each household.
We consider that each household's scheduler is synchronised to operate with the same discrete time steps of length T . At the start of a time step, should there be a task that has newly arrived into an empty queue (q m n = 1), the naive strategy of inaction would have the task start immediately such that t m n = 0. This corresponds with the conventional day-to-day scenario where a household appliance turns on immediately when the user demands so. However, in this paper we consider a setting where each household has an intelligent scheduler that is capable of acting by delaying each appliance's starting time by a continuous time period a m n ∈ [0, T ]. The joint action of all users is thus: Once an appliance has started executing a task from its queue, q m n is shortened accordingly. We consider that the appliance is uninterruptible and operates with a constant power consumption P m n . If an appliance is in operation for any length of time during a given time step, we consider its operational state to be x m n = 1. Otherwise, x m n = 0. Let d m n (k) denote the length of time that an appliance m is in operation during time step k for household n. The energy consumption of a household E n (k) during time step k is thus: On the grid side, we consider a dynamic energy price p(k) that is a linear function of the Peak-Average-Ratio [9], [25] of the energy consumption in the previous κ time steps: The reward signal then sent to each household is a weighted sum of a cost objective and a soft constraint. r n (k) = −r c,n (k) + wr e,n (k) (6) The first component of the reward function r c,n is the monetary cost incurred by each household at the end of each time step. The negative sign of the term (7) in (6) encourages the policy to delay energy consumption until the price is sufficiently low.
The second component of the reward is a soft constraint r e,n tunable by weight w. The soft constraint encourages the policy to schedule household appliances at a rate that matches the average rate of task arrival into the queue.
We adopt a practical assumption that each household can only observe its own internal state along with the published price at the previous time step, and receive its own cost incurred as the reward signal. For each household n, the system is thus partially observed We note that the state transition for all households can be considered to be Markovian because the transition to the next overall system state is dependent on the current state and the actions of all household agents: T : S × A 1 × · · · × A n → S . In contrast, we recall from (5) and (6) that the price of energy, and consequently each agent's reward, is a function of not only the current state and actions, but also of the history of previous states. Our pricing-based energy demand management process is, therefore, a generalisation of the Markov game formulation [26].

IV. SEMI-DISTRIBUTED DEEP REINFORCEMENT LEARNING FOR TASK SCHEDULING
The DSM microgrid problem that we introduce in the previous section presents various constraints and difficulties in solving it: (i) each household can choose its action based only on its local observation and the last published price, (ii) the reward function is dependent on the states and actions beyond the last timestep, (iii) the state and observation spaces are continuous, (iv) the household agents have no communication channels between them, and (v) a model of the environment is not available. Constraint (iv) is related to the general problem of environmental non-stationarity: concurrent training of all agents would cause the environmental dynamics to constantly change from the perspective of a single agent, thus violating Markov assumptions.
Requirements (i) and (ii) and environmental non-stationarity mean that a value-based algorithm such as Q-learning is not suitable, because these algorithms depend on the Markov assumption that the state transition and reward functions are dependent only on the state and action of a single agent at the last time step. Furthermore, requirement (v) rules out model-based algorithms. Thus, in this section, we propose an extension of the actor-critic algorithm Proximal Policy Optimisation (PPO) [22] for the multi-agent setting which we will refer to as Multi-Agent PPO (MAPPO).
Let π = {π 1 , π 2 , ..., π N } be the set of policies, parameterised by θ = {θ 1 , θ 2 , ..., θ N }, for each of the N agents in the microgrid. The objective function that we seek to maximise is the expected sum of discounted reward J(θ) = E s∼p π ,a∼π θ [ K k=i γ k−i r k ], where r k is obtained based on (6). In canonical policy gradient algorithms such as A2C, gradient ascent is performed to improve the policy: ∇ θn J n (θ n ) = E s∼p π ,an∼πn [∇ θn log π n (s, a n )Q πn n (s, a n )] .
(9) This policy gradient estimate is known to have high variance. It has been shown that this problem is exacerbated in multi-agent settings [19]. For this reason, we propose the implementation of PPO as it limits the size of each policy update through the use of a surrogate objective function. Additionally, we augment the individually trained actor networks with a central critic networkV π that approximates the value function for each household V π n based on the overall system state, thus receiving information for training of cooperative actions by individual households. The objective function we propose is the following: where ρ k (θ n ) = πn(a k |o k ) π n,old(a k |o k ) , and both and λ are hyperparameters.Â π n (k) is the estimated advantage value, which is computed using the critic network: A π n (s, a n ) ≈ r(s, a n ) + γV π (s(k + 1)) n −V π (s(k)) n . (11) We fit the critic network, parameterised by φ, with the following loss function using gradient descent: where the target value y(k) = r n (k)+γV π (s(k+1)) n . As the notation suggests, this update is based on the assumption that the policies of the other agents, and hence their values, remain the same. In theory, the target value must be re-calculated every time the critic network is updated. However, in practice, we take a few gradient steps at each iteration of the algorithm. The schematic of the proposed actor-critic framework is illustrated in Figure 1. The centralised criticnetwork itself utilises a branched neural network architecture shown in Figure 2a. To compute the estimated state value of a particular household n, V n , we provide the following input vector to the centralised criticnetwork: [ s n s 1 s 2 ... s 1 s n−1 s n+1 ...s N ] . The critic network would thus provide an estimate of the quantity V π (s) n = k E π θ [r(s n , a n )|s 1 , s 2 , . . . , s n−1 , s n+1 , . . . , s N ]. We compare the MAPPO algorithm to a similar algorithm that uses a decentralised critic-network instead. The architecture we utilise for the decentralised critic-network is shown in Figure 2b. The main consideration for such a design is to maintain a similar number of learnable parameters across all Fig. 1: The high-level architecture of the reinforcement learning setup with a centralised critic. Each actor and the critic is implemented as a feedforward multi-layer perceptron.
the N decentralised critic networks to the centralised critic, on the assumption that this would result in a similar computation resource requirements.

V. EXPERIMENTS
In this section, we present the settings and environmental parameters that we use for experimentation, evaluate our chosen reinforcement learning method against non-learning agents and the offline Energy Consumption Scheduler (ECS) method by [9], examine the effect of augmenting local observations in proposed system model, and consider the effects of household agents learning shared policies.

A. System Setup
We present experiments on a simulated system of identical households, each with 5 household appliances. As mentioned, their parameters relating to power consumption P m n , task arrival rates p m n (k) and lengths of operation per use λ m n are chosen to be a loose match to data sampled from the Smart* 2017 dataset [24], and thus resemble typical household usage patterns.
The period for each time step T is chosen to be 0.5 hours to achieve sufficient resolution, while limiting the amount of computation required for each step of 'real time' in the simulation. To reasonably limit the variance of trajectories generated, the maximum length of each simulation episode is limited to 240 time steps or 5 days. For each iteration of the training algorithm, 5040 time steps of simulation experience is accumulated for each batch. Each experiment is run with a number of random seeds; the extent of their effect is shown in the range of the lightly shaded regions on either side of the lines in the graphs shown.
For this chosen experimental setup, we run two baseline performance benchmarks that were non-learning: (i) Firstly, an agent that assigned delay actions of zero for all time steps. This represents the default condition in current households where the user dictates the exact time that all appliances turn on. (ii) A second baseline agent randomly samples delay actions from a uniform distribution with range [0, T ]. We compare these non-learning agents to agents trained using our proposed Multi-Agent PPO (MAPPO) algorithm as described in Section IV.
The results of the above-mentioned agents are compared in Figures 3a and 3b. The y-axis for both graphs shows the average reward gained and average cost incurred by all households for each day within each batch of samples. The x-axis shows the iteration index in the actor-critic training process. (Note that each batch consists of 105 days of simulation.) All trained policies demonstrate a sharp improvement in reward achieved from the onset of training. The monetary cost decreases with increasing reward, showing that the reward function is suitably chosen for the objective of minimising cost.
In Figure 4, we plot the average total energy consumed by the entire microgrid system for each half-hour time step. Note that the x-axis shows the wall-clock in the simulated environment using the 24 hour format. The blue line shows the energy-time consumption history if all home appliance were to be run immediately on demand; it reveals the peaks in demand in the morning and evening, presumably before and after the home occupants are out to work, and lows in demand in the night time. The uniform-random-delay agent dampens these peaks in demand by shifting a number of tasks forward into the following time steps (note the lag in energy consumption compared to the zero-delay agent). In contrast, the agent trained using reinforcement learning is able to not just delay the morning peak demand, but spread it out across the afternoon. The result is a lower PAR, and hence a lower monetary cost per unit energy.
Although the morning peak in energy demand is smoothed out by the MAPPO trained households, the same effect is less pronounced for the evening peak. We hypothesise that this was due to the training procedure: the batch of experience gathered in each iteration of the algorithm began at 00:00 hrs, and terminated at the end of a 24 hour cycle, providing fewer opportunities for the agents to learn that the evening demand can be delayed to the early hours of the subsequent day. Moreover, delaying tasks on the last cycle of each batch would have led to a lower value of soft-constraint reward r e,n , but an un-materialised cost improvement (higher r c,n ) for the subsequent morning (because of the midnight cut-off).  The results discussed above are significant for our particular application because it shows that while each household is controlled by different policies, all households are able to learn a cooperative strategy that flattens energy demand and consequently decrease the average cost for all users. The ability of the agents to learn in a decentralised manner is compatible with a microgrid system where there may be limited bandwidth to both receive and transmit information centrally. The individual user, or a small cluster of users, could improve their policy using local computing resources, while receiving only the output signal from the centralised critic network. Conversely, if computing resources for neural network training are only available centrally, update gradients can be transmitted to individual actor networks and result in similar performance.

B. Comparison with ECS
We compare our proposed MAPPO algorithm-based method to the Energy Consumption Scheduler (ECS) method proposed by [9], which is also a DSM-based solution. Instead of making decisions online as with MAPPO, ECS requires the following information in advance to schedule all energy consumption for the next day: (i) the total planned energy consumption for each household appliance for the entire day, and (ii) the mathematical function used to determine the price of electricity for a given time period. For a better comparison, we adapted the experimental setup to match the method used for ECS in its original paper: we considered a smart grid system with N = 10 households, split each day into H = 24 equal time intervals, fixed the length of operation of each task such that l m n = 1/λ m n , and used a quadratic price function as described in (13). We also consider different values of the multiplier w for the cost constraint in the reward function (6).
Input (i) required by the ECS method is thus computed as follows: In Figure 5, we plot the average aggregate energy consumed by the microgrid for each hour-long time step. Once again, the agents trained using reinforcement learning are able to delay the peaks in demand and spread the energy demand across subsequent time steps. The MAPPO agents trained with a coefficient of w = 2.2 consume almost the same energy over the entire day as the naive agents with zero delay; this constitutes a shift in energy demand that results in a decrease in the cost of energy.
In comparison to the MAPPO agents, ECS produces a flatter energy demand profile. Because the ECS algorithm receives both the energy consumption requirements and the price function before the day commences, it is able to schedule appliances to operate in the early hours of the day to exploit lower prices. However, since the energy consumption requirements received by ECS are average values, each household experiences a mismatch between its actual energy demand for the day and the schedule produced by ECS. Consequently, we compute a reward according to (6), where the value r c,n is taken to be the actual energy demand that is fulfilled by the ECS-generated schedule. The results plotted in Figure 6 show that the MAPPO agents with w = 2.2 achieve an average daily reward that is more than 10% higher than that achieved by ECS.
The results reveal the MAPPO method's key advantage, in that it can be utilised in a fully online scenario without requiring the price function or the total daily consumption in advance. It is able to rely on instantaneous observations of the local state and the price signal to determine how tasks should be scheduled by delaying them. This online approach and reliance on mainly locally observable information is more consistent with the nature of domestic electricity use.
Another advantage of MAPPO is that, in addition to shifting the demand of energy, we are able to implement energy reduction by tuning the coefficient w. Figure 5 shows that the MAPPO agents trained with a smaller coefficient of w = 0.4 exhibit lower energy demand than both the MAPPO agents with w = 2.2 and the ECS method.

C. Comparison with Methods using Decentralised Critics
Figures 3a and 3b also compare our proposed MAPPO algorithm to a similar PPO-based algorithm with decentralised critics, and the more traditional Advantage Actor-Critic (A2C) [27] algorithm. The results show that all the agents trained with deep reinforcement learning perform better than the nonlearning agents.
Our proposed MAPPO algorithm achieves the most sampleefficient learning speed and best performance, while the performance of the decentralised PPO algorithm is close. This demonstrates that good performance can also be achieved by learning in a completely decentralised manner. This would be useful in scenarios where connectivity of the smart meters outside of the household is limited, but local computing power can be utilised.
The canonical A2C algorithm is also able to reach the same level of performance, although it learns less efficiently with the same number of samples. Experiments with higher learning rates showed instabilities that manifested in exploding gradients -this was in spite of policy-based algorithms such as A2C being generally known to be more stable than valuebased algorithms. The reason is that, in our proposed microgrid environment, the updates to the policies for other households under A2C have a large effect on the overall system dynamics, possibly worsening the non-stationarity of the environment as perceived by each household. In contrast, the nature of the actor network update in the PPO algorithm limits the change in the actions taken by the updated policy, thus leading to stable performance increments.

D. Effect of augmenting local observations
We perform experiments to determine the necessity of including the simulation clock time and the previous price as part of the observations o n perceived by each agent. In an 8-household microgrid system, we train MAPPO agents that use observations without the time in one experiment, and train agents with the time but without the previous price in another experiment. The results are shown in Figure 7.
The agents trained without knowledge of the simulation clock time achieve a performance that significantly overlaps with the agents trained with the full observation vector. On the other hand, depriving the agents of knowledge of the price of energy on the previous time step shows both a marked decrease in average reward accrued and an increase in the average cost of electricity. The reason is that knowledge of the price at the previous time step is useful because it indirectly reveals information about the strategy used by the other household agents. In making this information observable, the priceaugmented observation ameliorates the problem of perceived environmental non-stationarity. In contrast, knowledge of the simulation clock time may be less useful because features within the local state, such as the appliance status x n and queue lengths q n , already provide time-related queues on how high the current demand for energy is.

E. Effect of Policy Sharing
We consider a scenario where groups of household agents may share a common policy. This may be useful when computing resources are unavailable in individual households, but accessible from a more centralised location. Instead of having each household agent controlled by a single policy, we conduct experiments where each policy is shared by two specific household agents. In these setups, each household always follows the same policy network.
The results are shown in Figure 7. While the reward achieved is similar, the systems where agents shared policies show a slightly lower resulting cost in energy. This is consistent with the findings of [19], which reported that sharing "policy ensembles" resulted in better scores. However, it should be noted that in the mentioned study, the policies are assigned to randomly chosen agents at each episode.
We investigated further by counting both the number of tasks generated in the household appliance queues q m n and the number of tasks done in each time step to compute the percentage of tasks done by each agent for each episode. Figure 8 shows the percentage of tasks that were completed by household 1 throughout the training process. It reveals that the shared policies are able to achieve a lower energy cost by strategically delaying tasks in the queue (which decreases energy consumption). On the other hand, single policies are unable to employ the same task-delaying strategy without overly sacrificing the second component of the reward that encourages task completion.

VI. CONCLUSION
In this paper, we proposed and implemented a smart grid environment, as well as a multi-agent extension of a deep actorcritic algorithm for training decentralised household agents to schedule household appliances with the aim of minimising the cost of energy under a real-time pricing scheme. Our  approach allows for an online approach to household appliance scheduling, requiring only the local observation and a price signal from the previous time step to act. A joint critic is learned to coordinate training centrally.
Our results show that our proposed algorithm is able to train agents that achieve a lower cost and flatter energy-time profile than non-learning agents. Our algorithm also achieves quicker learning than independent agents trained using the canonical Advantage Actor-Critic (A2C) algorithm. The successful results show that the choice of the observed variables and reward function design in our proposed environment contained sufficient information for learning.
Future work can involve extension of the smart grid environment to include more features and real-world complexities; we identify two such possibilities. Firstly, the architecture of the microgrid can be expanded to include more energy sources, which may be operated by producers or households. Secondly, we may expand the overall grid to include more agents and more types of agents, such as energy producers and brokers.