{'fraction_non_alphanumeric': 0.038367699449422435, 'fraction_numerical': 0.007643312101910828, 'mean_word_length': 4.231672879249972, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Liquid State Machine Learning for Resource and Cache Management in LTE-U Unmanned Aerial Vehicle (UAV) Networks

In this paper, the problem of joint caching and resource allocation is investigated for a network of cache-enabled unmanned aerial vehicles (UAVs) that service wireless ground users over the LTE licensed and unlicensed (LTE-U) bands. The considered model focuses on users that can access both licensed and unlicensed bands while receiving contents from either the cache units at the UAVs directly or via content server-UAV-user links. This problem is formulated as an optimization problem which jointly incorporates user association, spectrum allocation, and content caching. To solve this problem, a distributed algorithm based on the machine learning framework of liquid state machine (LSM) is proposed. Using the proposed LSM algorithm, the cloud can predict the users' content request distribution while having only limited information on the network's and users' states. The proposed algorithm also enables the UAVs to autonomously choose the optimal resource allocation strategies that maximize the number of users with stable queues depending on the network states. Based on the users' association and content request distributions, the optimal contents that need to be cached at UAVs as well as the optimal resource allocation are derived. Simulation results using real datasets show that the proposed approach yields up to 33.3% and 50.3% gains, respectively, in terms of the number of users that have stable queues compared to two baseline algorithms: Q-learning with cache and Q-learning without cache. The results also show that LSM significantly improves the convergence time of up to 33.3% compared to conventional learning algorithms such as Q-learning.


I. INTRODUCTION
The use of aerial wireless communication platforms carried by unmanned aerial vehicles (UAVs) is seen as a promising approach to improve the coverage and capacity of future wireless networks [2]. UAVs can have three key functions: aerial base stations, aerial relays, and user equipments. Due to their flying nature, UAVs can be used as aerial base stations to provide lineof-sight (LoS) connections toward ground users and, thus, potentially improving the rate, delay, and overall performance of wireless networks. Moreover, UAVs can be used as user equipments [3] connected to the cellular network for information dissemination and data collection. In addition, UAVs can act as relays between a source and a destination for scenarios in which a LoS link does not exist or for backhaul connectivity of base stations. However, deploying UAVs for wireless communication purposes faces many challenges [4]- [15] that include air-to-ground channel modeling, optimal deployment, energy efficiency, path planning, resource management, and security.

A. Related Work
The existing literature in [4]- [15] has studied a number of problems related to UAV communications and networking. For instance, in [4]- [6], the deployment of an unmanned aerial vehicle acting as a wireless base station that provide coverage for ground users is analyzed.
The authors in [7] proposed a statistical propagation model for predicting the air-to-ground path loss between a low altitude platform and a ground user. In [8], the authors investigated a multitier drone-enabled network complementing a terrestrial heterogeneous network. The work in [9] studied the use of LTE-Unlicensed (LTE-U) technology to enhance the achievable broadband throughput of UAVs. The authors in [10] studied the behavior of air-to-ground mmWave bands for UAVs using ray tracing simulations. In [11], the authors investigated the problem of efficient power control in UAV-supported ultra dense networks. The work in [12] studied the throughput maximization problem in mobile relaying systems by optimizing the source/relay transmit power along with the relay trajectory. Meanwhile, in [13], the authors jointly considered the radio resource allocation, three-dimensional placement, and user association of UAVs in the internet of things networks. The authors in [14] provided a comprehensive survey on available air-toground channels for UAV communication. In [15], the authors developed a new framework to optimize the minimum throughput of ground users by optimizing the multiuser communication scheduling and association jointly with the UAVs' trajectory and power control. However, most of these existing works [4]- [15] are focused on the direct communication links between UAVs and ground users and do not account for the ground-to-air fronthaul communications between the core network and the UAVs. Indeed, capacity-constrained fronthaul links will significantly limit the transmission rate of the links from the UAVs to the ground users. In order to reduce the traffic load on the fronthaul links and improve the performance of UAV-based communication, one promising approach is to cache popular content [16]- [18] at the UAVs thus allowing them to directly transmit data to the ground users without using wireless fronthaul transmissions, whenever possible [19]- [21].
To further improve performance and overcome the spectrum scarcity problem, the UAVs can be also equipped with LTE over the unlicensed band capabilities thus allowing them to use both licensed and unlicensed spectrum to service their ground users. Recently, there has been significant interest in studying the performance of LTE-U enabled cellular networks such as in [9], [22]- [27]. However, the LTE-U works in [22]- [27] do not consider the use of LTE-U with UAV-carried base stations. Meanwhile, even though the work in [9] considers LTE-U resource allocation in a UAV network, this work does not consider the effect of the limited-capacity cloud-UAVs links. In addition, the work in [9] does not exploit the use of caching at the UAV side.
Caching the popular contents at UAVs can help overcome the limited capacity of the cloud-UAVs links.

B. Contributions
The main contribution of this paper is a novel resource allocation framework that allows cache-enabled UAVs to effectively service ground users over licensed and unlicensed bands in a cloud network under fronthaul capacity constraints. To our best knowledge, this is the first work to jointly consider the joint use of the caching and LTE-U for UAV-assisted communication. Our main contributions include: • We jointly analyze the problems of resource allocation over licensed and unlicensed bands, user association, and caching content replacement. We formulate the problem as an optimization problem whose goal is to maximize the number of the stable queue users.
• To solve this resource allocation problem, we propose a novel self-organizing, decentralized algorithm to maximize the number of stable queue users. Unlike previous studies [4], [6]-4 [15], that overlook the limited-capacity of the UAV-cloud links, we propose a novel learning approach based on the powerful framework of liquid state machine (LSM) [28] that can jointly perform caching and resource allocation in a network of cache-enabled LTE-U UAVs.
The use of LSM enables the cloud to quickly learn the users' content request distribution so as to determine the content caching strategy for each UAV. It also enables the UAVs to autonomously adjust their spectrum allocation schemes to service users over licensed and unlicensed bands.
• Within the proposed LSM algorithm, we propose a search-based algorithm that can find the optimal resource allocation and derive optimal contents to cache so as to calculate the number of stable queue users. We also prove that the proposed search-based algorithm can find the optimal resource allocation for a given user association.
• Simulation results show that the proposed approach yields gains of up to 33.3% and 50.3% in terms of the number of users having a stable queue compared to Q-learning with cache and Q-learning without cache, respectively.
The rest of this paper is organized as follows. The system model and problem formulation are described in Section II. The LSM-based algorithms for content request distribution prediction and resource allocation are proposed in Section III. In Section IV, numerical simulation results are presented and analyzed. Finally, conclusions are drawn in Section V.

II. SYSTEM MODEL AND PROBLEM FORMULATION
Consider the downlink of an LTE-U network composed of a set K of K UAVs that service a set U of U LTE-U users and W WiFi access points (WAPs) that only service N w WiFi users, as shown in Fig. 1. In this model, the UAVs are equipped with cache storage units and can be deployed to act as flying cache-enabled LTE-U base stations to serve a set U of U ground users. The UAVs are controlled by a cloud-based server. Here, we consider dual mode cache-enabled UAVs that are able to access both the licensed and unlicensed bands. Each UAV can allocate at most one type of resource (i.e., licensed or unlicensed resources) to each user. The transmissions from the cloud to the UAVs occur over wireless fronthaul links using the licensed cellular band.
In this system, a frequency division duplexing (FDD) mode is considered for LTE on the licensed band. The FDD mode separates the licensed band for the downlink LTE-U users. A time division duplexing (TDD) mode with duty cycle method [29] is considered for LTE-U. Using the duty cycle method, the UAVs can use a discontinuous, duty-cycle transmission pattern so as to guarantee the transmission rate of WiFi users. In this case, the unlicensed band time slots will be divided between LTE-U and WiFi users. In particular, LTE-U transmits for a fraction ϑ of time and will be muted for a fraction 1 − ϑ of time which is allocated for WiFi transmission.
The WAPs transmit using a standard carrier sense multiple access with collision avoidance (CSMA/CA) protocol and its corresponding RTS/CTS access mechanism [30].
In our model, all ground users will request contents of equal size L from a set N of N contents that are stored at a cloud-based content server. The content request distribution of each user is p i = [p i1 , . . . , p iN ], where p ij represents the probability that user i requests content j. This distribution can be obtained by the machine learning algorithm that we will detail in Section III.
Each UAV k is equipped with a storage unit that can store a set C k of C popular contents that the users can request. If properly done, caching at the UAVs can significantly offload the fronthaul traffic of UAVs since each UAV can directly transmit its stored contents to the users without using fronthaul links. Hereinafter, caching at the UAVs is referred to as "UAV cache".
The cached contents at a UAV are assumed to be periodically refreshed at off peak hours when the UAVs return to their docking stations. Table I provides a summary of the notations used hereinafter.

A. WiFi data rate analysis
For the WiFi network, we assume that the WAPs will adopt a CSMA/CA scheme with binary slotted exponential backoff. Therefore, the saturation capacity of N w users sharing the same unlicensed band can be expressed by [10]: where P tr (N w ) = 1 − (1 − τ ) Nw with P tr (N w ) being the probability that there is at least one transmission in a time slot and τ being the transmission probability of each user. P s (N w ) = T c and T s are computed as done in [10].
We assume that one LTE time slot consists of T W WiFi time slots. Based on the duty cycle mechanism, the UAVs can occupy ϑ fraction of T W time slot on the unlicensed band while the WiFi users can occupy a fraction (1 − ϑ) fraction of T W time slots as shown in Fig. 2. Thus, the per WiFi user rate can be given by: where N w is the number of WiFi users on the unlicensed band. Given the rate requirement of each WiFi user γ, the fraction of the time slot on the unlicensed band allocated to the LTE-U users can be given by ϑ ≤ 1 − N w γ/R(N w ).

B. UAV data rate analysis
Next, we define the data rate of each user associated with any given UAV. The UAVs' content transmission link consists of the wireless fronthaul links that connect each UAV to the cloud (ground-to-air links) and the UAV-users links (air-to-ground links). As done in [4], [6], [7], we consider probabilistic LoS and non-line-of-sight (NLoS) links over the licensed band for both the UAVs' fronthaul links and UAV-user links. In such a model, NLoS links experience higher attenuation than LoS links due to the shadowing and diffraction loss.

1) UAVs-users links over the licensed band:
The LoS and NLoS path loss from UAV k to user i will be given by (in dB) [7]: where 20 log (d ki f 4π/c) is the free space path loss with d ki being the distance between user i and UAV k, f being the carrier frequency, and c being the speed of light. η l LoS and η l NLoS represent, respectively, additional attenuation factors due to the LoS/NLoS connections over the licensed band. As shown in [4], [6], [7], the probability of LoS connection depends on the environment, density and height of buildings, the locations of the user and the UAV, and the elevation angle between the user and the UAV. The LoS probability is given by [7]: where X and Y are constants which depend on the environment (rural, urban, dense urban, or others) and φ ki = sin −1 (h k /d ki ) is the elevation angle. Consequently, the average path loss from UAV k to user i will be given by [7]: where Pr l NLoS Based on the path loss, the downlink rate of user i associated with UAV k on the licensed band at time t will be: where F l is the downlink bandwidth on the licensed band, P K is the transmit power of each UAV, h i is the channel gain between user i and the cloud, and P C is the transmit power of the cloud over the fronthaul. σ 2 is the power of the Gaussian noise. Finally, u ki (t) is the fraction of the downlink licensed band allocated from UAV k to user i at time t with i u ki (t) = 1.

2) UAVs-users links over the unlicensed band:
In the considered model, the UAVs can only use the unlicensed band of the WiFi networks whenever the WiFi users' rate requirement is satisfied. Based on (2), we obtain a fraction ϑ of a time slot over the unlicensed band that can be occupied by UAVs. Therefore, the downlink rate of user i associated with UAV k on the unlicensed band can be given by: wherel u ki is the average path loss over the unlicensed band, F u is the bandwidth of the unlicensed band, and e ki (t) is the fraction of ϑ with i e ki (t) = 1.

3) Cloud-UAVs ground-to-air links:
The LoS and NLoS path loss from the cloud to UAV k can be given by [6]: L NLoS where ς is the additional path loss of the NLoS connection and d Ck is the distance between UAV k and the cloud. The average path lossL k of the fronthaul link of UAV k can be computed using (3) and (4). Here, we assume that the total bandwidth of the UAVs' fronthaul is F C which is equally divided among the users that received the contents from the cloud. Therefore, the fronthaul rate of each user that is associated with UAV k and requests the contents from the cloud is: where U C (t) is the number of the users that receive a content from the cloud at time t. U C (t) can be calculated by the content server as the users request contents from the content server.

C. Queueing model
Let V i (t) be the random content arrival (number of bits) for user i from the content server at the end of time slot t. We assume that each user can request at most one content during each time slot t and, consequently, V i (t) ∈ {0, L}. The queue length (i.e., number of bits) Q i (t) of user i at the beginning of time slot t, will be given by [31]: where R ki (t) is the rate of user i. Since the content transmission links include: (a) UAV-user on the licensed band, (b) UAV-user on the unlicensed, (c) cloud-UAV-user on the unlicensed band, and (d) cloud-UAV-user on the licensed band as shown in Fig. 1, the rate of content transmission from UAV k to user i will be given by: where the rate expression of link (c) is obtained from the fact that the time duration of a single data packet transmitted from the cloud to UAV k is 1/R Ck (t) and a single data packet transmitted from UAV k to user i is 1/R uki (t). Therefore, the data rate of the transmission from the cloud to user i is From (11), we can see that the rate of any given user that receives a content from the UAV cache (link (a) and link (b)) is larger than the rate of another user that receives contents from the cloud (link (c) and link (d)). We use the notion of queue stability to measure the users' content transmission delay. In essence, a queue Q i (t) is said to be rate stable if [31]: From [31, Theorem 2.8], we can also see that the queue

D. Problem formulation
Given this system model, our goal is to develop an effective spectrum allocation scheme for cache-enabled UAVs that can allocate appropriate bandwidth over the licensed and unlicensed bands to satisfy the queue stability requirement of each user. To achieve this goal, we formulate an optimization problem whose objective is to maximize the number of users that have stable queues. This maximization problem involves finding the optimal association U k for each UAV k, bandwidth allocation indicators on the licensed band u k , time slot indicators on the unlicensed band e k , and the set of cached contents C k for each UAV k. Therefore, this problem can be formalized as follows: The optimization problem in (13) is challenging to solve, because spectrum allocation and content caching depend on the user association which, in turn, depends on the rate of each user.
In fact, this problem can be shown to be combinatorial and non-convex, thus it is difficult to solve it using conventional optimization algorithms. Moreover, each UAV may not know the users' content requests which makes it challenging to determine which content to cache at the UAVs. To address these challenges, we propose a novel liquid state machine learning approach [32] to predict the users' content request distribution and perform resource allocation.
Liquid state machine is a novel kind of spiking neural networks [32] that are randomly generated. Learning algorithms based on LSM can store the users' behavioral information and track the state of a network over time. Therefore, an LSM-based algorithm will enable the cloud to leverage information on the users' behavior, that are stored in LSM, to predict the content request distribution and automatically adapt spectrum allocation to the any changes in the states of the network. Consequently, LSM-based algorithms are promising candidates for content request distribution prediction and wireless resource allocation problems.
Next, we first introduce the components of the proposed LSM algorithm. Then, we explain the entire process of using LSM to predict the users' content request distributions and to solve the problem in (13).

A. LSM Components
As illustrated in Fig. 3, an LSM-based algorithm consists of five components: a) agents, b) input, c) output, d) liquid model, and e) output function. Since the prediction of the content request distribution and resource allocation are function-specific, we design the specific components for the problems of predicting content request distribution and resource allocation, separately.

1) Content request distribution prediction:
The content request distribution prediction algorithm has the following components: • Agent: In the studied system, the LSM agent is the cloud. Since each LSM approach performs a content request distribution prediction for just one user, the cloud must implement U LSM algorithms.
• Input: The input of the LSM prediction algorithm for each user i is defined by a vector x iNx ] T that captures the context information related to user i's content request.
Such information includes age, gender, occupation, and device type (e.g., tablet or smartphone).
N x is the number of properties that constitute the context information of user i. The vector x i is used to predict the content request distribution y i of user i.
• Output: The LSM output is a vector of probabilities y i = [p i1 ,p i2 , . . . ,p iN ] that represents the discrete probability density function of the content requests of user i withp in being the prediction of p in that is the probability of user i requesting content n.
• Liquid model: A liquid model for each user i can store the users' dynamic features that are extracted from the users' context over time. These dynamic features can be used with the output functions to predict the users' behavior such as content request and mobility patterns.
Here, the liquid model consists of W 1 × W 2 × W 3 leaky integrate and fire neurons that are arranged in a 3D-column. In particular, each neuron that consists of resting state S, action state, and refractory period T f (ms) is defined as a excitatory neuron while the neuron that consists of resting state, inhibitory state, and refractory period is defined as an inhibitory neuron. The resting state indicates that the neuron does not receive any users' context information. The action state indicates that the neuron receives a certain amount of users' context and transmits this context to the connected neurons. The inhibitory state indicates that the users' information in the neuron is decreasing and, thus, cannot reach an action state. When each neuron transmits information to other connected neurons, its users' information may decrease to a value that is below that of the resting state. In this case, the neuron will have a refractory period during which the neuron returns to the resting state. The action state of neuron j of LSM for user i, v ij (τ ) at time τ can be given by [32]: where Z is the neuron resistance, I i (τ − 1) is the input of the users' information, and ρ is the neuron time constant. Based on (14), the LSM state can be given by where N W = W 1 × W 2 × W 3 is the number of neurons. Note that, the time duration of τ is much smaller than the time duration of t.
The connections from the input to the liquid model are made with probability P IN . The probability connection between neurons i and j can be given by: where ς ∈ {ς EE , ς EI , ς IE , ς II } is a constant that depends on the type of both neurons. In particular, ς EE denotes an excitatory-excitatory connection, ς EI is an excitatory-inhibitory connection, ς IE is an inhibitory-excitatory connection, and ς II is a inhibitory-inhibitory connection. d (i, j) is the Euclidean distance between neurons i and j. λ is a parameter that influences how often neurons are connected.
• Output function: The output function is used to build the relationship between the state of the LSM model and the prediction of each user's content request distribution. Let f i ∈ R N W ×N be the output function of LSM, where N is the total number of contents. In order to predict the user's content request distribution, f i is trained in an offline manner using ridge regression to approximate the prediction function: where is the LSM state sequence for user i. Here, y T,i is the target output of the LSM algorithm, I is an identity matrix, and δ is the learning rate. Based on the trained output function, the prediction of user i's content request distribution can be given by: Given the above components, the LSM based algorithm first uses (14) to store the users' context information and, then, uses (16) to train the output function of the LSM. After training, the LSM algorithm can predict the users' content request distribution based on the users' context information.
2) Resource allocation: The LSM reinforcement learning algorithm for resource allocation has the following components: • Agent: The agents in the LSM algorithm used for resource allocation are the UAVs.
where π k (t) = π k,a k1 (t), . . . , π k,a kA k (t) is a probability distribution for UAV k. If action a kj can maximize the average number of stable queue users, then π k,a kj = 1 − ε + ε A k , π k,a kj = ε A k , otherwise. A k is the number of actions that each UAV k can take and U k is the number of the users that can associate with UAV k which will be specified in Theorem 3. In particular, each UAV's action represents a UAV-user association scheme.
represents the resource allocation results. b kj (t) is the expected number of stable queue users when UAV k uses user association scheme j, which is: where A −k is the action set of all UAVs other than UAV k and b kj,a −k is the number of stable queue users as UAV k uses action a kj and the other UAVs use actions a −k . π −k,a −k (t) = n∈K,n =k, π n,an (t) is the marginal probability distribution over the action set of UAV k. To calculate b kj,a −k (a kj , a −k ), UAV k needs to determine the optimal contents to cache as well as the resource allocation for any given user association a kj . Based on the content request distribution y i obtained by the LSM-based prediction algorithm mentioned before, the optimal contents stored at UAV cache for a given user association can be determined by the following theorem: Theorem 1. Given U k , the user association of each UAV k, y i , the content request distribution of user i, and N C i , the number of contents that user i requests, the optimal contents stored at each UAV k can be given by: where C k is the set of contents that are stored at the cache of UAV k.
Proof. The contents stored at the cache are used to reduce the content transmission over fronthaul links from the cloud to UAV k. From (9), we can see that the bandwidth of the fronthaul is equally allocated to those users that request contents from the cloud and, hence, the objective of the optimal content caching policy is to minimize the number of the users that request contents from the cloud. For each user i, the average number of the requested contents is wherep in N C i is the average number of requests of content n. Therefore, the total average number of content requests for the users associated with UAV k is i∈U k . Clearly, UAV k will choose the contents that have the highest average number of requests to store at cache which can be given by max C k n∈C,i∈U k p in N C i . This completes the proof.
From Theorem 1, we can see that the optimal caching policy depends only on the users' association policies and their content request distributions. When the user association and cached contents [33] are determined, the transmission link for each content (i.e., from cloud or UAV cache) is determined. U kc (U kk ) is defined as the set of users whose requested contents are (not) stored at cache of UAV k. Given a determined user association a kj , (13) for each UAV k can be simplified as: The first term in (20) indicates that the users' requested contents are transmitted from the cache of the UAV. The second term indicates that the requested contents are transmitted from the cloud.
To solve the problem in (20), we propose a search-based algorithm. Then, we prove that the solution of the proposed algorithm is an optimal solution for (20). We define the vector u R k (t) = u R k1 (t) , . . . , u R kU k (t) with each element being the fraction of the licensed spectrum that UAV k uses to guarantee stable queues for its associated users where u R is the licensed spectrum that UAV k uses to guarantee a stable queue for user i. Similarly, e R k (t) = e R k1 (t) , . . . , e R kU k (t) is a vector with each element being the fraction of the unlicensed spectrum that UAV k uses to guarantee stable queues for the users where e R ki (t) = V i (t) R uki (1) is the fraction of the unlicensed spectrum that UAV k uses to guarantee a stable queue for user i.
is defined as a vector with each element u MR ki (t) being the minimum licensed bandwidth that UAV k needs to guarantee a stable queue for user i where From (21), we can see that, as the licensed spectrum that UAV k uses to guarantee a stable queue for user i is smaller than the unlicensed spectrum that UAV k needs to guarantee a stable queue for user i (u R ki (t) < e R ki (t)), u MR ki (t) = u R ki (t). Similarly, e MR k (t) = e MR k1 (t) , . . . , e MR kU k (t) is defined as a vector with each element e MR ki (t) being the minimum unlicensed bandwidth that UAV k uses to guarantee a stable queue for user i with We also define e MR k,max (t) = arg max will directly determine the number of searches needed by the search-based algorithm. We definẽ Givenũ R k e MR k,max (t) , we defineũ R k,max e MR k,max (t) = arg max . Based on these defined vectors, the search-based algorithm is given in Algorithm 1. The solution of Algorithm 1 is an optimal solution of the problem in (20) and can be characterized by the following theorem: , and e MR k,max (t) as well as N u MR k,max (t) and N e MR k,max (t) , the optimal resource allocation vectors for UAV k, u * (t) and e * (t) can be given by: where n (x) represents the number of elements in the support of vector x.
iii) If N e MR k,max (t) > 1, then [e * k , u * k ] = arg max n e MR k,max (t) + n ũ R k e MR k,max (t) .

Algorithm 1 Search-based algorithm
and e MR k (t) based on (21) and (22).
Traverse all u MR k,max (t) and modify e MR k,max (t) to find the maximum nonzero elements in u MR k,max (t) and e MR k,max (t).
Traverse all e MR k,max (t) and modify u MR k,max (t) to find the maximum nonzero elements in u MR k,max (t) and e MR k,max (t).

9: end if
Proof. For case i), If i∈U k u MR ki (t) ≤ 1 and i∈U k e MR ki (t) ≤ 1, then UAV k can guarantee stable queues for all of its users. Therefore, u * k (t) = u MR k (t), e * k (t) = e MR k (t). For case ii), N u MR k,max (t) > 1 indicates that the optimal resource allocation vector u MR k,max (t) is not unique. Therefore, UAV k needs to traverse all of the vectors u MR k,max (t) and its corresponding vectorẽ R k u MR k,max (t) . In u MR k,max (t), as shown in (21), u R ki (t) < e R ki (t). Hence, when UAV k changes the band allocated to a user from the licensed band to the unlicensed band, it will use more unlicensed resources than licensed resources to guarantee a stable queue for this user. In consequence, when n u MR ′ k,max (t) = n u MR k,max (t) − 1, we have n ẽ R k u MR ′ k,max (t) ≤ n ẽ R k u MR k,max (t) + 1 where u MR ′ k,max (t) and u MR k,max (t) differ by only one element that pertains to the user whose spectrum allocation has changed. Then, we can obtain that n u MR ′ k,max (t) + n ẽ R k u MR ′ k,max (t) ≤ n u MR k,max (t) + n ẽ R k u MR k,max (t) . Therefore, arg max the optimal resource allocation for UAV k. Case iii) can be proved in a similar way to case ii). This completes the proof.
From Theorem 2, we can see that the complexity of the search-based algorithm depends on the number of vectors u MR k,max (t) or e MR k,max (t). Moreover, the resource allocation over the licensed band u RM k,max (t) is related to the resource allocation via the unlicensed bandẽ R k u MR k,max (t) . Note that, the optimal resource allocation for the problem in (20) is not unique. Theorem 2 provides a method to find an optimal resource allocation for a given user association but not to find all of the optimal resource allocations for a given user association. Based on Theorem 1 and the search-based algorithm, the optimal caching contents and resource allocation for a given user association a kj are determined and, hence, b kj,a −k (a kj , a −k ) in (18) can be calculated. Given b kj,a −k (a kj , a −k ), the output of the LSM resource allocation algorithm can be obtained. Given the input and output, the LSM algorithm can find the relationship between them using the liquid model and output function that are introduced next.
• Liquid model: The liquid model in the resource allocation algorithm is used to store the network state information including UAV-user association schemes and their corresponding output results. The liquid model consists of W α 1 × W α 2 × W α 3 number of leaky integrate and fire neurons that are arranged in a 3D-column. The generation of the resource allocation LSM model is similar to the one in the content request distribution prediction case.
• Output function: The output function is used to build the relationship between the UAVsuser association schemes and the number of users with a stable queue.
be the output function of UAV k, where N α W is the number of the neurons in the liquid model. To train f α k , a linear gradient descent approach can be used to derive the following update rule, where f α k,i is row i of f α k , δ α is the learning rate, e k,i (t) is the expected output, and v k (t) is the LSM state at time t. From (24), we can see that the construction of f α k depends on the number of actions of UAV k. Therefore, we first need to determine the number of actions for each UAV k which can be given in the following theorem: Theorem 3. Given the locations of UAVs and users, the number of actions for UAV k can be given by: where Proof. To calculate the number of actions of UAV k, we need to first determine the set of the users that can associate with UAV k. From (13), we can see that, for user i associated with UAV k, the maximum data rate R ki,max in (11) must be larger than or equal to L. Here, we only need to consider the maximum data rate R ki,max of links (c) and (d). This is due to the fact that when the data rate of links (c) or (d) meets the data rate requirement, the data rate of links (a) or (b) will also meet the data rate requirement. Hence, where R lki (1) and R uki (1) imply that all of the bandwidth of the licensed and unlicensed bands is allocated to user i (i.e., u ki (t) = 1 and e ki (t) = 1) and R Ck,max = F C log 2   1 + P CLk j∈K,j =k indicates that all of the fronthaul bandwidth F C is allocated to user i. User i can associate with UAV k when the data rate R Ck,max of links (c) or (d) meets the rate requirement. There- R lki (1)+R Ck,max can be simplified as Each user i ∈ U k can associate with UAV k or associate with other UAVs, and, hence, total number of actions (user association schemes) is 2 |U k | where |U k | represents the number of elements (users) in U k . This completes the proof.
From Theorem 3, we can see that, when the data rate of fronthaul R Ck is much larger than the data rate of the links from UAVs to users (R lk or R uk ), the number of actions for each UAV k only depends on R lk or R uk . In particular, the bandwidths of fronthaul, licensed, and unlicensed bands will significantly affect the number of the users that can associate with UAV k. Based on Theorem 3, the output function f α k can be trained using (24). After the training of the output function f α k , the estimated number of users that have a stable queue given user association a kj is:

B. LSM Algorithm for content prediction and spectrum allocation
Based on the components of the proposed LSM algorithms detailed in Section III-A, next, we explain the entire process of using LSM algorithms to solve the problem in (13). To solve the problem in (13), the cloud first predicts the content request distribution of each user using the proposed LSM-based prediction approach. Based on the users' content request distribution, each UAV k uses the LSM-based learning algorithm with ǫ-greedy mechanism [23] to find the optimal user association policy. At each iteration, each UAV k takes one action (users association) a kj based on the ǫ-greedy mechanism. Once the users association is determined, the Input: The set of users' context, xj (t), UAVs' input m k (t).
Initialization: The cloud generates the liquid model for each user.
Each UAV generates a liquid model based on (14) and (15).
1: Calculate the time slots L based on (2).

4:
Estimate the number of the users that are at stable queuing state using (26).

11:
Update the output weight matrix f α k,i (t) based on (24). 12: Update the input a k (t) according to the result of the users choosing UAVs.

13: end for
optimal contents stored at the cache of UAV k can be determined by Theorem 1 and the optimal spectrum allocation will also be determined by the search-based algorithm. Based on the user association, optimal resource allocation and optimal caching contents, each UAV k can calculate the average number of stable queue users b kj,a −k (a kj , a −k ) when UAV k uses action a kj . In

IV. SIMULATION RESULTS
In our simulations, the content request data that the LSM uses to train and predict content request distribution is obtained from Youku of China network video index 1 . We consider a circular we use the MATLAB LSM toolbox described in [32]. Other system parameters are listed in Table II. For comparison purpose, we use two baselines: • The first baseline is Q-learning algorithm in [34] with content caching, which we refer to as "Q-learning with cache". This Q-learning algorithm is used to find the optimal resource allocation and user association for each UAV. For this Q-learning algorithm, the state is set to the LSM's input m k , the actions of Q-learning are the actions defined in our LSM algorithm, and the reward function r (x j , a j ) is the expected number of stable queue users in (18). The content request distribution is predicted by the proposed LSM algorithm and the contents to be cached are determined by Theorem 1.
• The second baseline is the Q-learning algorithm in [34] without content caching, which we refer to as "Q-learning without cache". The settings of the Q-learning without content caching are similar to the Q-learning algorithm with content caching. However, in the Qlearning without content caching algorithm, all of the contents requested by the users are transmitted from the content server.
We also compare our LSM based prediction algorithm with the echo state network (ESN) learning algorithm in [19]. All statistical results are averaged over 5000 independent runs.
In Fig. 4, we show how the average number of stable queue users changes as the number of UAVs varies. In this figure, the heuristic search for resource allocation algorithm represents a comparison algorithm that uses heuristic search to find optimal resource allocation and the proposed LSM algorithm for prediction and user association. From Fig. 4, we can see that the number of stable queue users increases as the number of UAVs increases. This is due to the fact that increasing the number of UAVs provides more connection options for the users, and, thus, improves the number of stable queue users. Fig. 4 also shows that the proposed LSM algorithm can yield up to 33.3% and 50.3% gains in terms of the number of stable queue users compared to Q-learning algorithm with cache and Q-learning without cache, respectively, for a network with 5 UAVs. These gains stem from the fact that the proposed LSM algorithm can use the historical user information to find an user association scheme and predict the users' content request distribution to improve content caching. From Fig. 4, we can also see that the proposed LSM algorithm can almost reach the same gain as the heuristic search algorithm which proves that the proposed search-based algorithm can find the optimal resource allocation for each UAV based on Theorem 1. Fig. 5 shows the cumulative distribution function (CDF) for the data rate resulting from all of the considered schemes. In Fig. 5, we can see that the data rate of almost 100% of users resulting from all of the considered algorithms will be below 2 Mbps. This is due to the fact that, the data rate requirement of each user is 2 Mbps and, hence, whenever a given user's rate requirement is satisfied, the UAVs will allocate the licensed or unlicensed spectrums to other  users. Fig. 5 also shows that the proposed approach improves the CDF of up to 27.56% and 48% gains at a rate of 1.8 Mbps compared to Q-learning with cache and Q-learning without cache, respectively. These gains stem from the fact that the proposed algorithm can store and use historical user information related to user association to find a better solution compared to Q-learning with cache and Q-learning without cache.
In Fig. 6, we show the variations of two content request probabilities of an arbitrarily selected user during one day. From Fig. 6, we can see that the accuracy of the predictions of the proposed LSM algorithm is within less than 10% from the real content request probability. Since this gap does not affect the ranking of each content request probability, the cloud can find the optimal contents to cache using the proposed algorithm. From Fig. 6, we can also see that the proposed LSM algorithm can reduce 13.4% mean squared error compared to the ESN algorithm. This is due to the fact that the neurons in the LSM model are more dynamic than the neurons in the ESN.
In consequence, the LSM model can store more information to predict the users content request distribution compared to the ESN model. Fig. 6 also shows that the sum of the probabilities with which this user requests contents 1 and 2 exceeds 0.5 during each hour. This is because the user always requests a small number of contents. Fig. 7 shows the number of iterations needed till convergence for both the proposed approach and Q-learning with cache. In this figure, we can see that, as time elapses, the number of stable queue users increases until convergence to their final values. Fig. 7 also shows that the proposed approach needs 400 iterations to reach convergence and exhibits a considerable reduction of 33.3% less iterations compared to Q-learning with cache. This is due to the fact that the LSM algorithm stores the users' states.
In Fig. 8, we show the variations in the number of the users that are allocated to the licensed and unlicensed bands as the number of contents stored at UAV cache varies. From Fig. 8, we   can see that the number of users over the licensed and unlicensed bands increases as the number of cached contents increases. This is due to the fact that content caching offloads the traffic over the cloud-UAV links thus reducing the bandwidth needed for having stable queues at the users.   This is because as the number of cached contents increases, the traffic load over the fronthaul can be reduced. From Fig. 9, we can also see that the proposed algorithm can achieve up to 15.5% and 64% gains in terms of the number of stable queue users compared to Q learning with cache and Q-learning without cache as the number of the contents stored at each UAV is 5. This is due to the fact that the LSM can store the users' historical information related to the user association and use this information to find the optimal user association. Fig. 9 also shows that the proposed LSM algorithm can reach almost the same performance in terms of the number of stable queue users as the heuristic search algorithm. This proves that the proposed Theorem 1 can find the optimal contents stored at the UAV cache.
In Fig. 10, we show how the average number of stable queue users changes as the fraction of UAV can allocate more WiFi time slots to its associated users and, hence, each UAV can satisfy more users' data rate requirements. Fig. 10 also shows that the proposed algorithm can yield up to 12.5% and 35.7% gains in terms of the average number of stable queue users for a UAV can use 0.9 of WiFi time slots. The 12.5% gain is due to the fact that the proposed algorithm can use historical users information to find the optimal user association for each UAV. The 35.7% gain stems from the fact that the proposed algorithm can use caching technology to reduce the traffic load over the fronthaul and use historical information related to the user association to find the optimal user association. Fig. 11 shows how the data rates of transmission links change as the bandwidth of the fronthaul varies. In this figure, the user is randomly chosen from the simulation. From Fig. 11, we can see that, as the bandwidth of the fronthaul (cloud-to-UAV links) increases, the data rates of links (a) and (b) remain unchanged while the data rates of links (c) and (d) increase. This is due to the fact that the contents transmitted over links (c) or (d) need to use the fronthaul. However, links (a) and (b) are used to directly transmit contents from the cache of any given UAV to a user without using the fronthaul. As the bandwidth of the fronthaul increases, the data rate of the fronthaul increases and, hence, the data rates of links (c) and (d) increase. Fig. 11 also shows that the unlicensed band data rate of links (b) and (c) are higher than the licensed band data rate of links (a) and (d). This stems from the fact that the user is interfered by the fronthaul transmission heavily.
In Fig. 12, we show how the average number of searches needed to find the optimal resource allocation varies as the number of users increases. From Fig. 12, we can see that, as the number of users increases, the average number of searches needed to find the optimal resource allocation for both algorithms increases. This is because as the number of the users increases, the number of the users associated with each UAV increases and, hence, each UAV needs to search more times to find the optimal resource allocation. Fig. 12 also shows that the proposed search-based algorithm can achieve up to 28.5% gain in terms of the number of searches needed to find the optimal resource allocation. This gain stems from the fact that the proposed search based algorithm can adjust the searching method based on the users' data rate requirements.

V. CONCLUSION
In this paper, we have developed a novel framework that uses flying cache-enabled UAVs to provide service for users in an LTE-U system. We have formulated an optimization problem that seeks to maximize the number of stable queue users. To solve this problem, we have developed a novel algorithm based on the machine learning tools of liquid state network. The proposed prediction algorithm enables the cloud to predict each user's content request distribution and, thus, determine the UAV's cached contents. Using the proposed LSM resource allocation algorithm, each UAV can decide on its spectrum allocation scheme autonomously with limited information on the network state. Simulation results have shown that the proposed approach yields significant performance gains. Moreover, the results have also shown that the use of LSM can significantly improve convergence time when compared to Q-learning.