{'fraction_non_alphanumeric': 0.04126153846153846, 'fraction_numerical': 0.011969230769230768, 'mean_word_length': 4.40962050599201, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

This paper studies how to schedule wireless transmissions from sensors to estimate the states of multiple remote, dynamic processes. Sensors make observations of each of the processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. Such estimation problems routinely occur in large-scale Cyber-Physical Systems, especially when the dynamic systems (processes) involved are geographically separated. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant to decide which sensors have network access and which ones do not. To solve this scheduling problem, we formulate an associated Markov decision process (MDP). Further, we solve this MDP using a Deep Q-Network, a deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for randomly generated example scenarios.


I. INTRODUCTION
Cyber-physical systems (CPS) are systems built through integration of sensors, communication networks, controllers, dynamic (physical) processes and actuators. They are playing an increasingly important role in modern society, in areas such as energy, transportation, manufacturing, and healthcare. The realization of these systems face substantial challenges in solving problems arising in diverse disciplines, ranging from communications and control to computing [1], [2]. The scale of typical CPS such as smart-grids, vehicular traffic networks and smart factories is large. Within these settings, remote estimation of the states of dynamical systems is an integral part of networked control systems. Supporting estimation and control applications over wireless networks has posed considerable challenges for the operation of networks and the design of protocols [3]. Figure 1 illustrates a simple yet general architecture of a networked cyber-physical system. Some processes are observed by a number of (simple) sensors, sensor observations are sent via a shared wireless network to a gateway that computes state estimates and control updates, which are then sent via (either Fig. 1. Networked cyber-physical system example scenario the same or a different) wireless network back to the actuators, hopefully driving all the processes to desired states.
From a networking perspective, one challenge lies in scheduling transmissions from the sensors to the gateway and back from the gateway to the actuators, because of both the volatile nature of wireless channels and the need to carefully schedule transmissions over a shared medium [4], [5]. While such channels provide the opportunity for diversity, they also aggravate the dynamic scheduling problem: which channel should be assigned to which sensor (or actuator), and when? This is a particular challenge when the number N of sensors/actuators and the number M of channels become large. Typically, N is much larger than M , since the resources do not scale with the size of the system. This problem is made more difficult by the absence of explicit information -e.g., we may not know the channel states when taking such a scheduling decision as such information is expensive to obtain.
The problem of scheduling is further exacerbated by control requirements. Usually, scheduling between sensors and the gateway is done to allow for better estimation and control performance, which may be at odds with typical communications performance parameters such as waiting times, throughput, etc.
[6]- [9]. To obtain high performance designs, it is required to take into consideration the dynamic behaviour of the underlying processes and the resulting urgency for communication.
For example, a sensor may wish to communicate urgently a radical change in the process it is sensing. Such requirements cannot be decoupled from the volatile behaviour of the wireless channels. Preempting our results, in Section V we present numerical results which show that algorithms reducing waiting times may not always be the best scheduling strategy for control applications.
Sensor scheduling problems where a single dynamic process is observed by multiple sensors have been studied in e.g.
[10]- [14]. More recently, sensor scheduling problems where multiple processes are observed by different sensors has also been investigated [15]- [17]. In the case of single channel systems (M = 1), optimal sensor scheduling problems with and without considering packet drops have been previously studied in [17] and [15], respectively. For the case M > 1 and additionally with packet transmission length constraints, some structural results were derived in [16], however numerical results were only provided for the M = 1 case. The focus of the current paper is on the case M > 1, and in particular to provide computationally scalable methods for solving optimal sensor scheduling problems to tackle large N and M .
In this paper, we only consider the "sensor to gateway" scheduling problem for remote state estimation. It may be noted that the ideas presented herein may in addition be readily extended to schedule "controller to actuator" communications. In estimation problems, the goal is to reduce estimation error covariances. This is because the larger the trace of the error covariance matrix, the poorer is the estimation quality. We assume that the gateway is model-aware, i.e., it has full knowledge of process-dynamics observed by each sensor, to allow Kalman filter-type estimation algorithms to be run. This restriction can be relaxed by incorporating model estimation/system identification in our approach, yet this would detract from our key contribution. The model assumptions are made precise in Section II. We emphasise that we only assume knowledge about the sensing model. For the wireless channels, no such knowledge is assumed. To put it succinctly, estimation is done in a model-aware manner while scheduling is done in a model-free manner.
To approach the dynamic scheduling problem, a roundbased protocol is a natural idea: the gateway selects a subset M of N sensors which communicate the sensor readings to the gateway. The sensor readings are then used by the gateway to update its estimates. The scheduling decision could be informed by knowledge about the quality of the estimates as well as by conjectures about channel state and probability of success of transmitting the readings to the gateway. The scheduling process is a discrete-time process with actions resulting in stochastic outcomes including quality of estimates. In other words, we have to formulate an "associated" Markov decision process (MDP). The MDP associated with our scheduling problem is formally described in Section III along with the optimization goal for scheduling. In the same section, we also present conditions for stability of our overall system and discuss computational issues.
Typically, the scale of a CPS is large. For our scheduling problem, this leads to an associated MDP with large state and action spaces. Traditional reinforcement learning based algorithms cannot be used to solve such MDPs due to Bellman's curse of dimensionality [18]. Bellman's curse of dimensionality is overcome by the use of function approximations [19]. Deep Q-Network (DQN) [20], [21] is one such algorithm that has shown tremendous promise in solving large MDPs in a scalable, model-free manner. DQN is a Monte-Carlo style algorithm that does not require system knowledge (channel Finally, we investigate in Section V the performance of the resulting schedules using this deep Q-learning scheme for a number of scenarios. For each given scenario, the system (process) and channel parameters are randomly selected. We compare our deep Q-learning method with several other scheduling schemes, namely 1) random channel allocation, 2) round robin scheduling (similar to periodic schedules commonly studied in the control literature [12], [13]), 3) a greedy algorithm to reduce waiting time (which we call holding time in this paper), and 4) a greedy algorithm on the trace of the estimation error covariance matrix. We find that random and round robin scheduling generally perform poorly, and in fact may lead to instability in some scenarios. The greedy algorithms perform reasonably well, but are still significantly outperformed by our deep Q-learning approach.

A. Sensing model
A diagram of the system model is shown in Fig. 2. We consider N independent, linear, discrete-time processes where x i,k ∈ R nx i is the state (vector) of process i at time k, and the process noise w i,k is i.i.d. (in time) Gaussian with zero mean and covariance matrix W i ∈ R nx i ×nx i . Each process is measured by a sensor as where y i,k ∈ R ny i is the measurement (vector) of process i at time k, and the measurement noise v i,k is i.i.d. Gaussian with zero mean and covariance matrix V i ∈ R ny i ×ny i . The noise processes {w i,k } and {v j,k } are assumed to be mutually independent for all i and j.
We assume that each sensor has the computational capability to run a Kalman filter, i.e., each sensor i can compute local state estimates and estimation error covariance matrices 1 x s . The Kalman filter equations are computed recursively as (see e.g. [23]): We will assume that every pair (A i , C i ) is observable, and every pair ( Then, the steady-state value of P s i,k for k → ∞ exists for each sensor [23] and will be denoted by P i . To simplify the presentation, we will assume that the local Kalman filters at the sensors have reached steady state 2 , so that P s i,k = P i , ∀i = 1, . . . , N, ∀k.

B. Scheduling and channel model
The Each channel is modelled using the Gilbert-Elliott (or Markovian packet drop [24]) model, with p m P(γ m,k = 0|γ m,k−1 = 1), 1 In situations where channels experience packet drops, transmission of local state estimates from sensors to the gateway in general gives better estimation performance than transmission of raw measurements [22]. The situation where raw measurements are transmitted to the gateway can also be handled using the deep Q-learning technique. 2 Convergence to steady state in general occurs at an exponential rate [23].
and with the channels being independent of each other. We will not assume knowledge of the channel parameters p m , q m , m = 1, . . . , M at the scheduler.

C. Protocol assumptions
Scheduling is assumed to be done at the gateway, with the decisions a m,k fed back to the sensors. 3 To support this scheduling model, we work under the following protocol assumptions. The protocol is round-based. At the start of a round, the gateway makes its scheduling decision, choosing a subset of sensors to transmit in this round; for each sensor, a channel is chosen as well. "Channel" can here refer to any orthogonal communication channel, a typical example would be frequency bands (orthogonal or with sufficient guard spaces), but time slots or spatial streams are also conceivable. It is worth noting that our algorithm generalizes to general wireless channels; we use the orthogonal channel model for simplicity of presentation.
Once this decision is taken, the gateway informs these sensors that they are selected to transmit over a particular channel. We assume that this downlink transmission from gateway to sensor works without errors. We can justify this by using all M orthogonal, stochastically independent channels to transmit this signalling information downlink, resulting in an exponentially reduced error probability. Downlink performance can be further improved by coding across channels (rather than just simple repetition coding) and time (since signalling information is relatively small, time overhead can be invested).
If we are concerned about downlink errors, we can also absorb (via upper and lower bounds) these error probabilities into the channel's Markov chain's success probabilities 1 − p m and q m , since the readings from a sensor are only received if both the downlink and the uplink transmissions are successful. A more detailed consideration of error probabilities for uplink and downlink is left for further study.
After these channel assignments have been received by the sensors, they send their respective data (local state estimates) uplink to the gateway. We assume that all sensors have similar amounts of data to send or that a reasonably tight upper bound exists. Once uplink transmissions are complete, the next round begins.

D. Remote Estimation at Gateway
At the gateway, state estimates and estimation error covariances of each of the processes are computed similar to [22], [25], as follows: where the matrix operator h i (.) is defined as As mentioned in Section I, the gateway is assumed to have knowledge of the system parameters of each of the N processes, which allows (3) to be computed. As given by (3), the estimation at the gateway works as follows: If a transmission from sensor i is successfully received at time k, then the gateway's estimate of x i,k will be equal to sensor i's local estimatex s i,k of x i,k , with estimation error covariance P i,k equal to the steady state estimation error covariance P i . Whereas if no information from sensor i is received, then the previous estimatex i,k−1 is propagated tox i,k = A ixi,k−1 , following the dynamics of the process (1), with the estimation error covariance matrix updated as III. PROBLEM DESCRIPTION The gateway wishes to find a scheduling policy to minimize the average sum of the trace 4 of the estimation error covariance matrices across all sensors and all times. We will design a Markov decision process (MDP) to solve the associated sequential decision making problem, namely: We will assume that the channel allocations at time k can depend on namely the estimation error covariances and channel transmission outcomes at the previous time step, which is information that is available to the gateway. From (3) we see that P i,k is always of the form h n i (P i ) for some n ∈ N, where h n i (.) denotes the n-fold composition of h(.) given in (4) and therefore the channel allocations at time k can, equivalently, depend on which is of smaller dimension than (6), as each τ i,k−1 is scalar while each P i,k−1 is a matrix. Below we will describe more formally problem (5) as an MDP. 4 We take the trace in order to transform the error covariance matrix into a scalar quantity that can be easily compared, and is commonly done in the estimation and control literature, see e.g. [15]- [17]. Note also that for an error covariance matrix can be regarded as the mean squared error of the state estimatex i,k .

A. Formulation as a Markov Decision Process
State space: First we define the state space. From the discussion above, the vector (τ 1,k−1 , . . . , τ N,k−1 , γ 1,k−1 , . . . , γ M,k−1 ) can be regarded as the state 5  However, if the channel parameters p m , q m , m = 1, . . . , M were known, then the MDP transition probabilities P(s |s, a) of reaching the next state s given that action a was taken in state s, can be computed by making use of independence of the processes and the channels, as well as the following relations:

B. Stability Condition
We now give a sufficient condition on when the optimal solution to the MDP (5) has bounded average cost, expressed in terms of the process and channel parameters. Such a stability condition is important for reliable monitoring of all of the processes. We make the following assumption: ..,N ρ(A i ) and q max max m=1,...,M q m , where ρ(A i ) denotes the spectral radius of A i . We assume that Then we have: 5 Note that the state of the MDP is different from the states x i,k of the processes. From now on we will mostly use the word "state" to refer to the state of an MDP.
Theorem III.2. Under Assumption III.1, the optimal solution to the MDP (5) has bounded average cost.
Proof. In the case ρ max < 1, condition (8) is always satisfied. Indeed, in this case each process is stable (and hence E[x i,k x T i,k ] remains bounded at all times k) and so the MDP (5) has bounded average cost even when there are no sensor transmissions. Thus we concentrate on the case ρ max ≥ 1.
First assume a single channel system where only channel m * is available. Consider a suboptimal policy where at each time instant the sensor with the largest holding time is chosen to transmit, provided that this holding time is greater than some L > 2N [26]. Using an argument similar to the proof of Theorem 3 in [26], we can show that this policy has bounded average cost if where P L can be expressed as P L = n<N P(n successful transmissions in L time steps). The rest of the argument in Theorem 3 of [26] assumes i.i.d. packet dropping channels. To extend the argument to Markovian packet drops as considered in this paper, we make the following observation: Given that there are n successful transmissions, then there will be L − n failed transmissions in these L time steps. Of these L − n failed transmissions, at most n of them will have followed a successful transmission (or equivalently at least L − 2n of them will have followed a failed transmission). From this observation, we have In the first inequality in (10), the term (max(q m * , 1 − p m * )) n upper bounds the probability of having n successful transmissions, while the term (max(p m * , 1−q m * )) n (1−q m * ) L−2n upper bounds the probability of having L−n failed transmissions, with at least L−2n also having the previous transmission fail. The second inequality in (10) holds as L n ≤ L N −1 for all n < N if L > 2N .
Taking limits in (10) gives Then by Assumption III.1, the condition (9) can always be satisfied for L sufficiently large, and so the suboptimal policy has bounded average cost. Thus the MDP (5) with only the single channel m * has bounded optimal average cost. As using additional channels does not increase the optimal average cost, the result follows.
Remark III.3. For the case of a single process and a single Gilbert-Elliott channel (with transition parameters p and q), a necessary and sufficient condition for bounded expected estimation error covariance [27] is that q satisfies: The condition (8) can be regarded as an extension of (11) to multiple processes and multiple channels, and intuitively says that the overall system has bounded cost provided the best channel (in terms of having the largest q m ) can keep the expected estimation error covariance of the most unstable process (i.e. having the largest spectral radius) bounded.

C. Computational Issues
Considering first the case where the channel parameters p m , q m , m = 1, . . . , M are known, numerical solution of (5) using dynamic programming techniques (e.g. using policy or relative value iteration) is in principle possible, after truncating the countable state space N N × {0, 1} M to a finite state space. But in practice, even for relatively small N and M , the sizes of both the state and action spaces can still be considerable, making exact numerical solution infeasible. For the case M = 1 (and relatively small N in numerical computation), a similar average cost problem has been previously studied [15], [17]. For M > 1 and additionally also considering packet transmission length constraints, some structural results were derived in [16], however numerical results were only provided for the M = 1 case.
If the channel parameters p m , q m , m = 1, . . . , M , are unknown (and hence the MDP transition probabilities are also unknown), as is assumed in this paper, then standard dynamic programming approaches for solving MDPs cannot be used. In this case an approach to solving problem (5) is by using reinforcement learning techniques [19], which can be used to tackle problems in a model-free manner without knowledge of the transition probabilities. The well-known Q-learning algorithm will in principle converge to the optimal solution, but in general the convergence is rather slow and requires both the state and action spaces to be small in order for the method to be feasible [19], [28].
To overcome the above mentioned problems of large state space and unknown channel parameters, we will use Qlearning methods in combination with deep neural networks for function approximation. The deep Q-learning technique introduced in [20], [21] provides a scalable method for sensor scheduling without assuming any knowledge of channel parameters. We will describe how this technique can be applied to problem (5) in the next section.
Remark III.4. A scheduling problem for control of multiple processes using a deep reinforcement learning approach has also been recently studied by us in [29], but without consideration of packet drops. The setup in [29] also requires extra overhead in the transmission of error information (between the state estimates at the sensor and controller) from the sensors to the scheduler at every time step, which could be considerable.

IV. SENSOR SCHEDULING USING DEEP REINFORCEMENT LEARNING
Consider the discounted problem . a 1,k , . . . , a M,k all distinct (12) where α < 1 is a discount factor. In this paper we will approximate the solution to problem (5) by solving (12) using reinforcement learning techniques, with a discount factor α close to 1 [30]. While Q-learning type algorithms for average reward maximization problems exist [31], most reinforcement learning algorithms assume a discounted setting, in particular the deep reinforcement learning techniques of [20], [21]. A more formal justification for solving the discounted problem will be given in Section IV-A. Let us rewrite problem (12) as the equivalent average reward maximization problem: The Q-factor or action-value function Q(s, a) represents the expected future reward associated with taking action a when at state s [19], [28]. The Q-factor version of the Bellman equation for problem (13) is: where s represents the value of the next state given the current state s and action a, and Q * (., .) are the optimal Q-factors. For each given state s, we can deterministically find an optimal action a * (s) by a * (s) = argmax a Q * (s, a).
The Q-learning algorithm will, in principle, converge to the optimal Q-factors, but requires both the state and action spaces to be small. For large MDPs one can approximate Q * (s, a) by a function Q(s, a; θ) parameterized by a set of weights θ [19], and then learning these weights. Deep reinforcement learning refers to the case where the function approximation Q(s, a; θ) uses a (deep) neural network, which has been crucial in recent key breakthroughs in artificial intelligence such as in the playing of Go [32]. The deep Q-learning techniques introduced in [20], [21] also included a number of modifications to the basic Q-learning with function approximation approach, aimed at stabilizing the learning algorithm, in particular 1) the notion of experience replay, and 2) fixing the target Qnetwork at regular intervals. Using these ideas, our approach to solving problem (13) is given as Algorithm 1. In Algorithm 1, a t = (a 1,t , . . . , a M,t ) corresponds to the allocation of the M channels at time t, and r t = N i=1 −trP i,t . The state s t could be chosen as s t = (τ 1,t−1 , . . . , τ N,t−1 , γ 1,t−1 , . . . , γ M,t−1 ) as Algorithm 1 Deep Q-network for wireless sensor scheduling 1: Initialize replay memory D to capacity K 2: Initialize network Q with random weights θ 0 3: Initialize target networkQ with weights θ − = θ 0 4: Initialize s 0 5: for t = 0, 1, . . . , T do 6: With probability δ select a random action a t , otherwise select a t = argmax a Q(s t , a; θ t ) 7: Execute a t , and observe r t and s t+1 8: Store (s t , a t , r t , s t+1 ) in D 9: Sample random mini-batch of transitions (s j , a j , r j , s j+1 ) from D 10: Set z j = r j +α max a Q (s j+1 , a ; θ − ) for each sample in mini-batch 11: Perform a mini-batch gradient descent step on (z j − Q(s j , a j ; θ t )) 2 to obtain θ t+1

12:
Every c steps set θ − = θ t 13: end for in Section III-A, however for the simulations in Section V we will further augment the state to s t = (τ 1,t−1 , . . . , τ N,t−1 , tr(h 1 (P 1,t−1 )), . . . , tr(h N (P N,t−1 )), which we have found in some cases gives faster convergence for the algorithm. We note that Algorithm 1 can be run online, and is model-free in that it does not need knowledge of the channel parameters p m , q m , m = 1, . . . , M .
A. Deriving an associated discounted cost problem for scheduling As stated in Section III, the aim of the scheduler is to find a scheduling policy that minimizes the average estimation error covariances, i.e. solve an associated average cost problem. If the communication channels satisfy Assumption III.1, then it follows from Theorem III.2 that there exists a scheduling policy that ensures that the cost is bounded. As mentioned in Section III-B, such a stability property is key to facilitating reliable monitoring of all processes. In other words, in addition to helping minimize the average estimation error, the scheduler is also required to stabilize the estimator. However, it can be shown that stabilizability is a direct consequence of solving the average cost problem. Hence the scheduler merely needs to focus only on the average cost minimization problem, see (5) for details.
We are interested in a reinforcement learning solution to the scheduling problem. While there are many efficient algorithms to solve the infinite/finite horizon discounted case problem, there are very few algorithms to solve the infinite/finite horizon average case problem, see [28] and [31] for further details. In this subsection, we derive an associated discounted case problem and show that the policy found therein is an -optimal policy for the average case problem. Further, can be made arbitrarily small by controlling the discount factor, α, of the associated MDP. Let J k be the single-stage cost, at time k, associated with the estimation problem. Before proceeding, we state Abel's theorem for our setting [30].
Theorem IV.1 (Abel). Let {J k } k≥0 be a sequence of positive real numbers, then From Theorem III.2 it follows that there exist (stabilizing) scheduling policies with finite associated average costs. It now follows from Abel's theorem that: Further, given > 0 there exists an α( ) ≈ 1, dependent on , such that: In addition to , α( ) also depends on the actual realizations of the single stage cost sequences {J k } k≥0 . If one wishes to find an -optimal policy, then one can choose a discount factor α( ), provided the "orders" of these single stage costs are known. In our problem, the single stage costs are unbounded. However, it is clear that the discount factor α ↑ 1 as ↓ 0. Hence, in our experiments, we choose a discount factor very close to 1.
To summarize, we briefly discussed a recipe to derive an associated discounted cost problem for scheduling. Further, we also showed that solving this associated discounted cost problem leads to an -optimal scheduling policy.
. . , N matrices (for the given dimensions in this example C i are row vectors and V i are scalars) and channel transition probabilities p m , q m , m = 1, . . . , M are randomly generated. The eigenvalues of A i are drawn from the range (0, 1.3). The entries of C i are drawn from the range (0, 1), and W i and V i are generated by random orthogonal transformations of a diagonal matrix with random diagonal entries drawn from the range (0.2, 1.0). The channel transition probabilities p m and q m are randomly generated from the range (0, 1).
The following hyper-parameters for Algorithm 1 are used in our simulations. In the deep-Q network we use two hidden layers with each hidden layer having 1024 units, and a fully connected layer with outputs for each of the N P M = 120 actions. The discount factor is set to α = 0.95. The experience replay memory has size K = 20000. The exploration parameter δ in line 6 of Algorithm 1 is attenuated from 1 to 0.01 at the rate of 0.999, i.e. δ ← max(0.999δ, 0.01) after every iteration. In the neural network training (step 11 of Algorithm 1) the ADAM optimizer [33] is used with a learning rate of e −4 and a learning rate decay of 0.001. The size of each mini-batch is 32. The target Q-network is updated once every c = 100 time steps.
Algorithm 1 is run to train our deep Q-network. In order to get a better idea of the training quality over time, we will reset the process after each T = 500, which we will refer to as an episode [19]. Running on a standard Intel Core i7 4790 with 8 Gb RAM (without GPU), each episode of training when using the above hyper-parameters took around 30 seconds to complete. The empirical average cost over different episodes for one randomly generated set of parameters is plotted in Fig. 3.
After training for 200 episodes, simulating the process over 50000 time steps then gives an empirical average cost of around 17.8. We compare this performance with the following policies: 1) A random policy that at each time k randomly allocates M out of the N sensors to the M channels, 2) A round robin policy where M successive sensors (modulo N ) are randomly allocated the M channels at every time instance, 3) A greedy policy on the holding times, where at each time k we allocate the M sensors with the largest τ i,k−1 (in the case of ties we take the sensors with smallest indices) randomly to the M channels, 4) A greedy policy on the error covariance, where at each time k we allocate the M sensors with the largest trP i,k−1 randomly to the M channels. Simulation over 50000 time steps gives an empirical average cost of around 62.7 for the random policy, 42.7 for the round robin policy, 31.3 for the greedy policy on holding times, and 22.4 for the greedy policy on error covariances. The performances of the greedy policies are also shown in Fig. 3 for comparison. We see that our deep reinforcement learning approach consistently outperforms these greedy policies after around 40-50 episodes of training.
In Table I we report further comparisons between the random policy, round robin policy, greedy policies, and the performance using deep reinforcement learning, for 10 different randomly generated sets of parameters A i , C i , W i , V i , p m , q m (making sure that condition (8) is satisfied), while keeping N = 6 and M = 3. The same hyper-parameters for training the deep Q-network as in the above was used. We can see that the random policy and round robin policy generally do not perform well (although the performance of the round robin policy seems to be better than the purely random policy), and in fact appears to lead to instability in some of the scenarios. The greedy policy on the error covariances performs better than the greedy policy on the holding times, due to the use of more knowledge of the system parameters. We also see that in each scenario the approach using deep reinforcement learning performs significantly better than all the other considered policies.
Remark V.1. Existing non-control aware scheduling strategies include random, round robin, or greedy strategies with respect to a given parameter, which are also used to, e.g., reduce waiting/holding times. However, in estimation and control applications such strategies do not perform as well as strategies which take into account the dynamics of the processes, as can be seen in Table I.

VI. CONCLUSION
This paper has studied a sensor scheduling problem for allocating wireless channels to sensors for the purposes of remote state estimation of dynamical systems. With the aim of providing a scalable method which can handle larger state and action spaces than previous work in the literature, we have proposed an approach to the sensor scheduling problem based on deep reinforcement learning. The resulting scheduling algorithm can be run online, and is model-free with respect to the wireless channel parameters. Numerical results have demonstrated that our approach consistently outperforms other suboptimal sensor scheduling policies. Future work will include the control scheduling problem where the gateway communicates control updates back to the actuators, and how these transmissions can be scheduled.