{'fraction_non_alphanumeric': 0.035984666986104454, 'fraction_numerical': 0.008481073310972687, 'mean_word_length': 4.558189081225033, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
IRS-Assisted Ambient Backscatter Communications Utilizing Deep Reinforcement Learning

We consider an ambient backscatter communication (AmBC) system aided by an intelligent reflecting surface (IRS). The optimization of the IRS to assist AmBC is extremely difficult when there is no prior channel knowledge, for which no design solutions are currently available. We utilize a deep reinforcement learning-based framework to jointly optimize the IRS and reader beamforming, with no knowledge of the channels or ambient signal. We show that the proposed framework can facilitate effective AmBC communication with a detection performance comparable to several benchmarks under full channel knowledge.


I. INTRODUCTION
Ambient backscatter communication (AmBC) is a key enabler for energy-efficient networking in the Internet of Things, where AmBC devices (or tags) convey information on top of modulated radiofrequency (RF) signals [1]. Despite this, a key problem is that the backscattered information signal at the reader experiences severe direct-link interference (DLI) from the ambient signal, which may be unknown and orders of magnitude stronger, resulting in poor detection performance.
Various methods, from transceiver design to channel estimation and modifications to network infrastructure, have been proposed to improve the AmBC detection performance. Work in [2] proposed a tag which performs frequency shift modulation to separate the backscattered signal from the DLI, at the cost of tag complexity. Machine learning-inspired approaches were presented in [3], [4]. Recently, the use of an intelligent reflecting surface (IRS) to improve AmBC performance was also explored in [5], [6].
IRSs have received significant research interest due to their ability to impose variable phase shifts on impinging signals to obtain desired reception at a receiver [7], [8]. The joint phase shift optimization of a large number of reflectors allow favorable received signal strength scaling, proportional to the IRS area [9]. A further advantage of reconfigurable phase shifts is the ability to focus signals in different directions to reduce inter-user interference [10]. This makes IRS an ideal candidate to address the problem of severe DLI in AmBC systems, and hence improve AmBC detection.
In this letter, we study the signaling design and phase shift optimization of an IRS-assisted AmBC system in indoor settings [11] such as smart homes, to improve the detection performance. A passive IRS is used for energy efficiency. Typically, IRS optimization requires full channel state information (CSI). However, the AmBC component poses severe challenges to CSI acquisition: a) channels between all nodes The authors are with the School of Engineering, The Australian National University, Canberra, Australia (e-mail: xiaolun.jia@anu.edu.au). exist in typical AmBC systems, but the channels involving the ambient source are often unknown and its signal varies every sample; b) as the tag is also a passive reflector, it is difficult to resolve the IRS-tag channels. For these reasons, many channels are very hard to estimate, and thus the CSI is assumed to be unavailable, rendering IRS optimization a formidable challenge. Our work presents, to our best knowledge, the first solution to optimizing the IRS under these realistic AmBC conditions. Work in [12] is the only other study on IRS optimization in AmBC systems, but nonetheless adopted a full CSI assumption; while work in [3] optimized the AmBC detection without CSI, but in a system with no IRS. We perform the optimization with a deep reinforcement learning (DRL) approach using the deep deterministic policy gradient (DDPG) algorithm. DRL can solve highly complex problems involving IRSs such as the joint optimization of phase shifts and other parameters [13], and more generally, problems with large numbers of control variables and only partially observable environments [14]. We utilize AmBC domain knowledge to propose a modified DDPG algorithm compared to those in conventional IRS works with known CSI [13], [15], which works off individual signal samples. Our results show that the proposed approach performs comparably to several full-CSI benchmarks.

II. SYSTEM MODEL AND PROBLEM FORMULATION
Consider an AmBC system in Fig. 1 with an ambient RF source, a single-antenna tag, an IRS with N reflectors and a reader with M antennas. Hereafter, we assign subscripts S, T , I and R to the source, tag, IRS and reader, respectively. We adopt a discrete-time signal model, as in [16]. The source transmits signal s[ ], whose samples are i.i.d. and follow CN (0, P S ), where P S is the transmit power. The assumption of a complex Gaussian ambient signal is commonly adopted in AmBC systems [3], [16], [17].
The tag is equipped with two impedances and performs onoff keying modulation, with the data symbols being either 0 or 1. For ease of exposition, we assume that the tag has arXiv:2103.07083v2 [cs.IT] 26 Jul 2021 a built-in battery, which powers the circuit operation over a typical lifetime of several years. As the tag is a diffuse scatterer, we take the strengths of signal paths undergoing two or more reflections at the tag before reaching the reader to be negligible. However, this assumption does not apply at the IRS, due to its ability to enhance the overall signal strength while balancing between different reflection links.
Linear combining is performed at the reader using the vector g ∈ C M ×1 , where we set g 2 = 1 without loss of generality.
Conventional energy detection [3] is utilized to recover the backscattered data symbols after applying the combiner. All channels are assumed to undergo frequency-flat quasistatic fading. The channels from the source to tag, source to IRS, source to reader, tag to reader, tag to IRS and IRS to reader are respectively. Moreover, we assume that the ambient source does not cooperate with the system, and that the reader has no knowledge of the ambient signal or the CSI of any channel in the system.
Each IRS reflector has a reconfigurable phase shift, denoted by θ n ∈ [0, 2π) for the n-th reflector. The incident signal at each reflector is subject to the reflection coefficient Θ n = e jθn arising from the phase shift. The amplitude scaling of all IRS reflectors is set to unity. Thus, the matrix of reflection coefficients at the IRS is Θ = diag(Θ 1 , . . . , Θ N ).
The signal received by the tag, consisting of the direct source-tag and reflected source-IRS-tag signal paths, is The tag backscatters where α denotes the tag splitting coefficient, which is set to 1 without loss of generality; and b[ ] denotes the -th sample of the backscattered data symbol. We assume that the duration of one backscatter symbol spans L samples. Therefore, denoting the k-th backscatter symbol as b k , we have The reader receives the ambient RF signal from the sourcereader and source-IRS-reader paths, plus the backscattered signals from four paths, as combinations of {source-tag, source-IRS-tag} multiplied by {tag-reader, tag-IRS-reader} paths: is the noise at the reader. The final signal after the combiner is given by g H y R . We adopt the shorthand h A h 0 , h I αh 1 and h AI h A + h I , where subscripts A and I represent the ambient and information-bearing components, respectively.
As per conventional AmBC systems (e.g., [17]), the energies of the received 0 and 1 symbols are modeled as Gaussian random variables, with means and variances given by µ0 , µ0 µ1 , which is the energy ratio between the symbol with the higher energy and the symbol with the lower energy. We note that the GRCD directly determines the BER of the AmBC system. Under a central limit theorem assumption and reasonably large L, the BER is derived from [3] as where Q N (·) is the Gaussian Q-function. One can show that the BER reduces as GRCD increases. Hence, we aim to maximize the GRCD of the IRS-assisted AmBC system (which minimizes the BER) by jointly designing the IRS and reader parameters in the following problem: Note that ∆ G in Problem (P) can only be obtained when the full CSI is available. When no CSI is available, we use instantaneous signal observations to estimate the "sample" GRCD for the t-th time step, denoted by ∆ (t) G . The sample GRCD is estimated using µ 0 and µ 1 averaged over L samples in each symbol. As our work assumes the absence of CSI, we propose a DRL-based solution in the next section, by approximating ∆ G based on instantaneous signal observations.

III. DRL-BASED FRAMEWORK A. Reinforcement Learning Fundamentals
Reinforcement learning problems concern the interactions between an agent and the environment in order to maximize a reward, which can be formulated as a Markov decision process (MDP). At time step t, the state s t characterizes the environment. Based on s t , the agent takes an action a t drawn from a policy π. The action influences the environment, which takes on a new state s t+1 ; while a reward r t is provided to the agent. The agent stores experiences over time, each of the form s t , a t , r t , s t+1 . The agent aims to determine an optimal policy π * : S → A to maximize the action-value function and thus the expected discounted reward (return), given by where γ ∈ [0, 1] denotes the discount factor. DDPG is an algorithm applicable to MDPs with continuous action spaces [14]. Two components, the actor and critic, simultaneously learn the policy and Q-functions, respectively. Each consists of two deep neural networks (DNNs), termed the training and target nets, as shown in Fig. 2.
Denote the parameters (weights) of the training and target actor nets by θ µ and θ µ , respectively, and those of the training and target critic nets by θ q and θ q , respectively. Here, we use the notation µ(s t ) instead of π to denote the policy, to highlight the continuous action space. The agent stores past experiences in its replay memory E. At each time step, the agent samples a minibatch of B experiences from E and computes target Q-values for each experience using where Q (s, a) is the Q-value from the target critic net. Next, gradient descent is performed on θ q to minimize the overall loss between the target Q-values and those produced by the training net, with the loss function given by Subsequently, the training actor net, which produces the agent's policy, is updated by sampling the policy gradient (PG), which provides an approximation of the policy's expected return. Maximizing the return involves performing gradient ascent, whose update rule is given by where α µ is the learning rate and the subsequent terms approximate the PG. Finally, to ensure stability during training, the target nets are updated every T up time steps according to where τ 1 is the update coefficient for the target nets.

B. DRL Problem Reformulation and Proposed Algorithm
Problem (P) can be formulated as an MDP, where the reader acts as the agent and is responsible for the joint design of the IRS reflection coefficients and its own combiner as its action. The reader runs the proposed DRL algorithm to achieve this design by interacting with the wireless propagation environment, which is characterized by the channels.
Each channel coherence period, consisting of T time steps, is defined as one episode. Due to the stochastic and unknown nature of the channels, each episode is independent from others, as the underlying environment varies with the channels in each episode. Thus, a key difference of the DRL framework in our work compared to existing DRL works involving IRS (e.g., [13], [15], [18]), where the CSI is known, is the fact that the actor and critic are trained in each episode based only on the observations in the current episode.
• State: The current state is the concatenation of the previous combiner and IRS reflection coefficients (2M + 2N elements). As existing DNN implementations do not support complex inputs, the real and imaginary components of combiner weights and reflection coefficients are used: • Action: The action space consists of the real and imaginary components of only the updated reflection coefficients (2N elements) based on the current state, for reasons which will be explained in the sequel. That is, • Reward: Instead of using ∆ (t) G directly as the reward, we modify the reward function to r t = 100(∆ (t) G − 1). From our experimentation, we observed that when random combiner weights and reflection coefficients were used, the majority of GRCD values were close to 1. This may lead to underfitting when different states and actions result in similar rewards. The multiplicative factor of 100 thus spreads out the reward space, such that each reward may be relatively more distinct, allowing faster convergence to be achieved by the critic nets. In our experimentation with various state and action spaces, we observed poor performance when the combiner and reflection coefficients were set together. Thus, inspired by [3], we propose to pre-set the combiner to the optimal eigenvector beamformer corresponding to the signal observations in the current DRL step, independent from the reflection coefficients (which are updated later). This has two practical advantages. First, the optimal combiner results in a relatively large GRCD (compared to e.g., a random combiner), which can be further improved through tuning the reflection coefficients. Second, the Q-function, which relies on both state and action, is derived from only one combiner, as opposed to one from the current state and one from the action (which may be vastly different), resulting in more effective fitting of the Q-function.
The T time steps in each episode are divided into training and data transmission phases. In the training phase, each step is a DRL agent-environment interaction, and consists of two pairs of backscatter pilot symbols, with each pair being a 0 followed by a 1, and each symbol spanning L t samples. The estimated channel covariance matrices for the first pilot pair, denoted by C i , i ∈ {0, 1}, are first obtained using the current reflection coefficients (i.e., Θ (t−1) ), and given by and are then refined using the maximum eigenvalue and corresponding eigenvector [3]. Then, we update the combiner by solving the equation C 1 g = λC 0 g, where λ represents the generalized eigenvalues of {C 0 , C 1 }. The combiner is set to the eigenvector of the maximum eigenvalue λ + if λ + > 1/λ − , with λ − being the minimum eigenvalue, and the eigenvector for λ − otherwise. The updated combiner is concatenated with the current IRS reflection coefficients to give an 'intermediate' state s t,int . We then feed this intermediate state into the actor to obtain the updated reflection coefficients, which is the action for this DRL step. The second pair of pilot symbols is then transmitted, yielding C i , i ∈ {0, 1}, and refined similarly. The sample GRCD for the current DRL step is then obtained by evaluating ∆ (t) As the underlying channels are unknown, we reserve the first T train time steps of each episode, where the agent explores while training the actor and critic to learn the current reward function. The T train steps are divided into two phases: the agent takes random actions for the first T 1 steps, followed by actions generated from the actor for T 2 steps. After T train steps, the final set of {g, Θ} is fixed for the remainder of the episode, which is the data transmission phase, where a shorter symbol duration of L d samples is used to provide a higher data rate. We note that the use of a random training phase is critical to achieve desirable results for this problem.
The DRL approach is presented in Algorithm 1. In each episode, the reader begins by instructing the tag to transmit a fixed number of pilot symbol pairs, followed by its message. Once initiated, the tag only needs to transmit its pilots and data, without further interactions with the system. A control link exists between the reader and the IRS for sending the phase shift instructions during each DRL step. The DDPG algorithm complexity is O(2T train B( Y −1 y=1 u y u y+1 )) per DNN per episode from forward and backward propagation, where u y is the number of hidden units in layer y. The computation of the covariance matrices incurs very small cost compared to forward and backward propagation; as such, the complexity per episode is similar to the DDPG variants in e.g., [13].

IV. NUMERICAL RESULTS
We demonstrate the performance of the DRL framework with the following simulation setup: all IRS channels undergo Rician fading as in [12] with Rician factor 3; while all other channels experience Rayleigh fading, typical of scattering occurring in a smart home environment. The path loss exponent is 2.5 for all channels. The ambient signal frequency is 2.4 GHz with transmit power P s = 20 dBm, typical of a Wi-Fi access point. The source, tag, IRS and reader are located at [−5, 0], [0, 0], [0, 5] and [5,0] m, respectively. The reader has M = 4 antennas and the noise variance is P w = −95 dBm. We group sub-groups of IRS reflectors to take on the same reflection coefficients for efficient computation [19], such that each of the N effective reflectors is one-wavelength-sized.
The training and target nets in the actor and critic are fullyconnected DNNs, with an input layer, two hidden layers and an output layer. The sizes of the actor and critic nets are    (14). With our values of {T 1 ,T 2 }, the training phase is < 20% of a typical channel coherence time of 100 ms [20] for backscatter setups with stationary nodes and a software-defined radiotype reader. In addition to the results from Algorithm 1, four benchmarks are included for comparison. These are: 1) the optimal zero-forcing (ZF) and 2) eigenvector (EIG) combiners when no IRS is present, 3) the ZF combiner with IRS, and 4) the EIG combiner with IRS, initialized using the g and Θ solutions from 3). All four benchmarks are obtained under full CSI and average noise power. Due to the large variance in GRCD values as a result of small-scale fading, the median GRCD is shown for each IRS size and benchmark. One should note that with no CSI on top of the varying ambient signal and noise, the DRL framework can never outperform the full-CSI benchmarks. However, Fig. 3(a) shows that the proposed CSIfree DRL framework still performs within 25% of the best benchmark under full CSI (i.e., Benchmark 4) for the range of N values. This is a significant result that illustrates the effectiveness of the proposed CSI-free framework. Fig. 3(b) translates the median GRCD into the equivalent BER in the data transmission phase using (5). We find that the BER achieved with Algorithm 1 is comparable with the best benchmark for all values of N . Moreover, one orderof-magnitude BER improvement may be achieved using a moderately-sized IRS with N = 64, which is a significant gain over the best non-IRS benchmark.  where N = 64. One may observe that the GRCD obtained under small L t is fairly poor, due to the inaccurate estimation of the channel covariance matrices based on few signal samples against a large number of channels. Thus, longer symbols are required for accurate estimation. Reasonable performance is achieved once L t becomes moderately large (e.g., L t = 100), beyond which diminishing returns are observed. These results suggest that a fairly accurate covariance estimation may be achieved when L t reaches a certain level compared to the total number of implicitly observed channels. It should be noted that long training symbols are needed only in the training phase. Once a satisfactory set of {g, Θ} is obtained, the data symbol duration can be reduced without affecting the GRCD. Fig. 4(b) shows the convergence quality of Algorithm 1, in terms of the effect of the random training phase length on the median GRCD. We find that the GRCD increases with longer training phases to a certain extent; while having T 1 = 0 results in very poor GRCD performance. As such, a random training phase is critical to mitigating the no-CSI nature of the problem. Note that for small N , T 1 may be reduced to shorten the training phase without much BER penalty.

V. CONCLUSION
In this letter, under unknown CSI and ambient signal, we proposed a DRL-based framework to optimize the design parameters of an IRS-assisted AmBC system based on actual signal observations. The lack of CSI and varying reward function in each episode were mitigated with independent training in each episode and zero discount factor. Moreover, we utilized the optimal eigenvector combiner without IRS as initialization in each DRL step for effective exploration. Our results indicated that the DRL framework is capable of similar performance compared to various full-CSI benchmarks. The design of the DRL framework to take advantage of timecorrelated channels is a useful avenue for further work.