{'fraction_non_alphanumeric': 0.04828611452585837, 'fraction_numerical': 0.012407444466680007, 'mean_word_length': 4.211561382598331, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptive Caching via Deep Reinforcement Learning

Caching is envisioned to play a critical role in next-generation content delivery infrastructure, cellular networks, and Internet architectures. By smartly storing the most popular contents at the storage-enabled network entities during off-peak demand instances, caching can benefit both network infrastructure as well as end users, during on-peak periods. In this context, distributing the limited storage capacity across network entities calls for decentralized caching schemes. Many practical caching systems involve a parent caching node connected to multiple leaf nodes to serve user file requests. To model the two-way interactive influence between caching decisions at the parent and leaf nodes, a reinforcement learning framework is put forth. To handle the large continuous state space, a scalable deep reinforcement learning approach is pursued. The novel approach relies on a deep Q-network to learn the Q-function, and thus the optimal caching policy, in an online fashion. Reinforcing the parent node with ability to learn-and-adapt to unknown policies of leaf nodes as well as spatio-temporal dynamic evolution of file requests, results in remarkable caching performance, as corroborated through numerical tests.


I. INTRODUCTION
In light of the tremendous growth of data traffic over both wireline and wireless communications, next-generation networks including future Internet architectures, content delivery infrastructure, and cellular networks stand in need of emerging technologies to meet the ever-increasing data demand. Recognized as an appealing solution is caching, which amounts to storing reusable contents in geographically distributed storageenabled network entities so that future requests for those contents can be served faster. The rationale is that unfavorable shocks of peak traffic periods can be smoothed by proactively storing 'anticipated' highly popular contents at those storage devices and during off-peak periods [24], [3]. Caching popular content is envisioned to achieve major savings in terms of energy, bandwidth, and cost, in addition to user satisfaction [24].
To fully unleash its potential, a content-agnostic caching entity has to rely on available observations to learn what and when to cache. Toward this objective, contemporary machine learning and artificial intelligence tools hold the promise to empower the next-generation networks with 'smart' caching control units (CCU), that can learn, track, and adapt to unknown dynamic environments, including space-time evolution of content popularities and network topology, as well as entityspecific caching policies. Deep neural networks have lately boosted the notion of "learning from data" with field-changing performance improvements reported in diverse machine learning and artificial intelligence tasks [14]. Deep neural networks (DNNs) can cope with the 'curse of dimensionality' by providing compact low-dimensional representations of high-dimensional input data [4], [39]. Combining deep learning with reinforcement learning [17], deep reinforcement learning has created the first artificial agents to achieve human-level performance across various challenging domains, including AlphaGo beating the world Go champion [21], machine translation [38], and intelligent transportation systems [6]. As another example, a DNN system was built to operate Google's data centers, and shown able to consistently achieve a 40% reduction in the amount of energy consumed for cooling [13]. The system provides a general-purpose framework to understand complex dynamics, which has been applied to address other challenges in the data center environment and beyond.

A. Prior art on caching
Early approaches to caching include the least recently used (LRU), least frequently used (LFU), first in first out (FIFO), random replacement (RR) policies, and their variants. Albeit simple, these schemes cannot deal with the dynamics of content popularities and network topologies. Recent efforts have gradually shifted toward developing learning and optimization based approaches, that can 'intelligently' manage the cache resources. For unknown yet time-invariant content popularities, multi-armed bandit online learning (e.g., [18]) was pursued in [5], while distributed, coded, and convexified generalizations were lately reported in [30], [15].
In realistic networks however, popularities exhibit dynamics, which motivate well the so-termed dynamic caching. A Poisson shot noise model was adopted to approximate the evolution of popularities in [34], for which an age-based caching solution was developed in [16]. Reinforcement learning based methods have been pursued in [28], [32], [29]. Specifically, a Q-learning based caching scheme was developed in [28] to model global and local content popularities as Markovian processes. Considering the Poisson shot noise popularity dynamics, a policy gradient reinforcement learning based caching scheme was devised in [32]. Assuming stationary file popularities and service costs, a dual-decomposition based Q-learning approach was pursued in [29]. However, those approaches entail discrete or quantized states, which prevents them from dealing with a practically large continuous state-action space.
On the other hand, the aforementioned contributions focus on developing caching policies for a single caching entity. A more common setting emerging in next-generation networks however, involves a network of interconnected caching nodes.
It has been shown that considering a network of connected caches jointly can considerably improve performance [20], [7]. For instance, leveraging network topology and the broadcast nature of links, the coded caching strategy in [20] further reduces data traffic over a network. This idea has been generalized in [25] to an online setting, where popularities are modeled Markov processes. Collaborative and distributed online learning approaches (e.g., [10], [7], [15], [36]) have also been pursued.
We identify the following challenges that need to be addressed when designing practical caching methods for nextgeneration networks. c1) Networked caching. Caching decisions of a node, in a network of caches, influences decisions of all other nodes. Thus, a desired caching policy must adapt to the network topology and policies of neighboring nodes. c2) Complex dynamics. Content popularities are random and exhibit unknown space-time, heterogeneous, and often non-stationary dynamics over the entire network. c3) Large continuous state space. Due to the shear size of available content, caching nodes, and possible realizations of content requests, the decision space is huge.

B. This work
Targeting these challenges, this paper considers a twolevel network caching where a parent node is connected to multiple leaf nodes to serve user file requests. Such a network indeed acts as the building block in many practical caching settings, including content delivery and 5G networks. We model the interaction between caching decisions of the parent and leaf nodes, along with the space-time evolution of file requests, from a reinforcement learning vantage point. Then, we reinforce the parent node with a policy, that can learn-and-adapt to space-time evolution of file requests as well as local policies used by leaf nodes. This policy just relies on a few observations from leaf nodes to make caching decisions. Nevertheless, the continuous state space and large number of actions challenge classical reinforcement learning algorithms. To address this challenge, a deep Q-network along with experience replay and target network is used to first approximate and then learn the so-called Q-function, hence the optimal policy in an online fashion.

II. MODELING AND PROBLEM STATEMENT
Consider a two-level network of interconnected caching nodes, where a parent node is connected to N leaf nodes, indexed by n ∈ N := {1, . . . , N }. The parent node is connected to the cloud through a (typically congested) backhaul link; see Fig. 1. One could consider this network as a part of a large hierarchical caching system, where the parent node is connected to a higher level caching node instead of the cloud; see Fig. 2. In a content delivery network for instance, edge servers (a.k.a. points of presence or PoPs) are the leaf nodes, and a fog server acts as the parent node. Likewise, (small) base stations in a 5G cellular network are the leaf nodes, while a serving gate way (S-GW) may be considered as the parent node; see also [9, p. 110].  All nodes in this network store files to serve file requests. Every leaf node serves its locally connected end users, by providing their requested files. If a requested content is locally available at a leaf node, the content will be served immediately at no cost. If it is not locally available due to limited caching capacity, the content will be fetched from its parent node, at a certain cost. Similarly, if the file is available at the parent node, it will be served to the leaf at no cost; otherwise, the file must be fetched from the cloud at a higher cost.
To mitigate the burden with local requests on the network, each leaf node stores 'anticipated' locally popular files. In addition, this paper considers that each parent node stores files to serve requests that are not locally served by leaf nodes. Since leaf nodes are closer to end users, they frequently receive file requests that exhibit rapid temporal evolution at a fast timescale. The parent node on the other hand, observes aggregate requests over a large number of users served by the N leaf nodes, which naturally exhibit smaller fluctuations and thus evolve at a slow timescale.
This motivated us to pursue a two-timescale approach to managing such a network of caching nodes. To that end, let τ = 1, 2, . . . denote the slow time intervals, each of which is further divided into T fast time slots indexed by t = 1, . . . , T ; see Fig. 3 for an illustration. Each fast time slot may be e.g., 1-2 minutes depending on the dynamics of local requests, while each slow time interval is a period of say 4-5 minutes. We assume that the network state remains unchanged during each fast time slot t, but can change from t to t + 1.
Consider a total of F files in the cloud, which are col-lected in the set F = {1, . . . , F }. At the beginning of each slot t, every leaf node n selects a subset of files in F to prefetch and store for possible use in this slot. To determine which files to store, every leaf node relies on a local caching policy function denoted by π n , to take (cache or nocache) action a a a n (t + 1, τ ) = π n (s s s n (t, τ )) at the beginning of slot t + 1, based on its state vector s s s n at the end of slot t. We assume this action takes a negligible amount of time relative to the slot duration; and define the state vector s s s n (t, τ ) := r r r n (t, τ ) := [r 1 n (t, τ ) · · · r F n (t, τ )] to collect the number of requests received at leaf node n for individual files over the duration of slot t on interval τ . Likewise, to serve file requests that have not been served by leaf nodes, the parent node takes action a a a 0 (τ ) to store files at the beginning of every interval τ , according to a certain policy π 0 . Again, as aggregation smooths out request fluctuations, the parent node observes slowly varying file requests, and can thus make caching decisions at a relatively slow timescale. In the next section, we present a two-timescale approach to managing such a network of caching nodes.

III. TWO-TIMESCALE PROBLEM FORMULATION
File transmission over any network link consumes resources, including e.g., energy, time, and bandwidth. Hence, serving any requested file that is not locally stored at a node, incurs a cost. Among possible choices, the present paper considers the following cost for node n ∈ N , at slot t + 1 of interval τ c c c n (π n (s s s n (t, τ )), r r r n (t+1, τ ), a a a 0 (τ )) :=r r r n (t+1, τ ) (1−a a a 0 (τ )) (1− a a a n (t+1, τ ))+ r r r n (t+1, τ ) (1− a a a n (t+1, τ )) (1) where c c c n (·) := [c 1 n (·) · · · c F n (·)] concatenates the cost for serving individual files per node n; symbol denotes entrywise vector multiplication; entries of a a a 0 and a a a n are either 1 (cache, hence no need to fetch), or, 0 (no-cache, hence fetch); and 1 stands for the all-one vector. Specifically, the second summand in (1) captures the cost of the parent node fetching files for end users, while the first summand corresponds to that of the parent fetching files from the cloud.
We model user file requests as Markov processes with unknown transition probabilities [28]. Per interval τ , a reasonable caching scheme for leaf node n ∈ N could entail minimizing the expected cumulative cost; that is, 1 c c c n (π n (s s s n (t, τ )), r r r n (t+1, τ ), a a a 0 (τ )) where Π n represents the set of all feasible policies for node n.
In particular, a reinforcement learning based approach using tabular Q-learning was pursued in our precursor [28], which can be employed here to tackle this fast timescale optimization. The remainder of this paper will thus be on designing the caching policy π 0 for the parent node, that can learn, track, and adapt to the leaf node policies as well as user file requests. At the beginning of every interval τ + 1, the parent node takes action to refresh its cache; see Fig. 4. To design a caching policy that can account for both leaf node policies and file popularity dynamics, the parent collects local information from all leaf nodes. At the end of interval τ , each leaf node n ∈ N first obtains the time-average state vector s s s n (τ ) := (1/T ) T t=1 s s s n (t, τ ), and subsequently forms and forwards the per-node vectors s s n (τ ) (1 − π n (s s s n (τ ))) to its parent node. This vector has nonzero entries the average number of file requests received during interval τ , and zero entries if π n stores the corresponding files. Using the latter, the parent node forms its 'weighted' state vector as where the weights {w n ≥ 0} control the influence of leaf nodes n ∈ N on the parent node's policy. Similarly, having received at the end of interval τ + 1 the slot-averaged costs c c c n (π 0 (s s s 0 (τ )); {a a a n (t, τ + 1), r r r n (t, τ + 1)} T t=1 ) c c c n (a a a n (t, τ + 1), r r r n (t, τ + 1) , a a a 0 (τ + 1)) (4) from all leaf nodes, the parent node determines its cost c c c 0 (s s s 0 (τ ), π 0 (s s s 0 (τ ))) w nc c c n (π π π 0 (s s s 0 (τ )); {a a a n (t, τ + 1), r r r n (t, τ + 1)} T t=1 )).
Having observed {s s s 0 (τ ), a a a 0 (τ ), c c c 0 (s s s 0 (τ − 1), a a a 0 (τ )} τ τ =1 , the objective is to take a well-thought-out action a a a 0 (τ + 1) for next interval. This is tantamount to finding an optimal policy function π * 0 to take a caching action a a a 0 (τ+1) = π * 0 (s s s 0 (τ )). Since the requests {r r r n (t, τ )} n,t are Markovian and present actions {a a a n (t, τ )} n,t also affect future costs, the optimal RL policy for the parent node will minimize the expected cumulative cost over all leaf nodes in the long term, namely where Π 0 represents the set of all feasible policies; expectation is over {r r r n (t, τ )} n,t , as well as (possibly random) parent a a a 0 (τ ) and leaf actions {a a a n (t, τ )} n,t ; and the discount factor γ ∈ [0, 1) trades off emphasis of current versus future costs. It is evident from (6) that the decision taken at a given state s s s 0 (τ ), namely a a a 0 (τ + 1) = π(s s s 0 (τ )), influences the next state s s s 0 (τ +1) through π n,τ (·) in (3), as well as the cost c c c 0 (·) in (5). Therefore, problem (6) is a discounted infinite time horizon Markov decision process (MDP). Finding the optimal policy of an MDP is NP-hard [23]. To cope with this complexity of solving (6), an adaptive reinforcement learning approach is pursued next.
IV. ADAPTIVE RL-BASED CACHING Reinforcement learning deals with action-taking policy function estimation in an environment with dynamically evolving states, so as to minimize a long-term cumulative cost. By interacting with the environment (through successive actions and observed states and costs), RL seeks a policy function (of states) to draw actions from, in order to minimize the average cumulative cost as in (6) [33]. Towards developing an RL solver of (6), define the so-termed value function to indicate the quality of policy π 0 , starting from initial state s s s 0 (0) as which represents the average cost of following policy π 0 to make caching decisions starting from s s s 0 (0). Upon finding V π0 (·) for all π 0 ∈ Π 0 , one can readily obtain π * 0 that minimizes V π0 (s s s 0 (0)) over all possible initial states s s s 0 (0).
For brevity, the time index and the subscript 0 referring to the parent node will be dropped whenever it is clear from the context. To find V π (·), one can rely on the Bellman equation, which basically relates the value of a policy at one state to values of the remaining states [33, p. 46]. To formalize this, define first the state transition probability from the current state s s s to the next state s s s upon taking action a a a as P a a a s s ss s s = Pr s s s(τ + 1) = s s s | s s s(τ ) = s s s, a a a = π(s s s) .
Leveraging (7) and (8), the Bellman equation for the value function V π (·) is given by where the average immediate cost can be found as With A (S) denoting the action (state) space, if P a a a s s ss s s were known ∀a a a ∈ A and ∀s s s, s s s ∈ S, then finding V π (·) would be equivalent to solving the system of linear equations (9). Indeed, if one could afford the complexity of evaluating V π (·) for all π ∈ Π 0 , the optimal policy π * is the one that minimizes the value function for all states. However, the large (possibly infinite) number of policies in practice discourages such an approach. An alternative is to employ the so-termed policy iteration algorithm [33, p. 64] outlined next. Define first the state-action value function, also called Q-function for policy π Q π (s s s, a a a) := E 1 c c c (s s s, a a a) + γ s s s P a a a s s ss s s V π (s s s ) .
This function captures the expected immediate cost of starting from state s s s, taking the first action to be a a a, and subsequently following policy π to take future actions onwards. The only difference between the value function in (7) and that of Q-function in (11) is that the former takes the first action according to policy π, while the latter starts with a a a as the first action, which may not necessarily be taken when adhering to π. Having defined the Q-function, we are ready to present the policy iteration algorithm, in which every iteration i entails the following two steps: Policy Evaluation. Find V π i (·) for the current policy π i by solving the system of linear equations in (9).
To perform policy evaluation, we rely on knowledge of P π i (s s s) s s ss s s . However, this is impractical in our setup that involves dynamic evolution of file requests, and unknown caching policies for the leaf nodes. This calls for approaches that target directly the optimal policy π * , without knowing P a a a s s ss s s , ∀a a a, s s s, s s s . One such approach is Q-learning [33, p. 107].
In the ensuing section, we first introduce a Q-learning based adaptive caching scheme, which is subsequently generalized in Section V by invoking a deep Q-network.

A. Q-learning based Adaptive Caching
The Q-learning algorithm finds the optimal policy π * by estimating Q π * (·, ·) 'on-the-fly.' It turns out that π * is the greedy policy over the optimal Q-function [33, p. 64], that is π * (s s s) = arg min α α α∈A Q π * (s s s, α α α), ∀s s s ∈ S where Q π * is estimated using Bellman's equation for the Qfunction. This is possible because the V -function is linked with the Q-function under π * through (see [33, p. 51] for details) V π * (s s s) = min α α α∈A Q π * (s s s, α α α), ∀s s s.
Substituting (14) into (11) which plays a key role in many reinforcement learning algorithms. Examples include Q-learning [37], and SARSA [26], where one relies on (15) to update estimates of the Qfunction in a stochastic manner. In particular, the Q-learning algorithm follows an exploration-exploitation procedure to take some action a a a in a given state s s s. Specifically, it chooses the action minimizing the current estimate of Q π * (·, ·) denoted byQ τ (·, ·), with probability (w.p.) 1− τ , or, it takes a random action a a a ∈ A otherwise; that is, a a a = arg min Take action a a a n (t, τ ) via π n 7 a a a n (t, τ ) = π n (s s s(t − 1, τ )), if t = 1 π n (s s s(T, τ − 1)), if t = 1 (16) The estimated Q-function for a single state-action pair is subsequently updated, by following a gradient descent step to minimize the loss in (16), which yields the updatê where β > 0 is some step size. Upon evaluating the gradient and merging terms, the update in (17)  For easy reference, the main steps of our two-timescale Qlearning based caching scheme are listed as Alg. 1.
Three remarks are worth making at this point. Remark 1. As far as the fast-timescale caching strategy of leaf nodes is concerned, multiple choices are possible, including e.g., LRU, LFU, FIFO, [12], the multi-armed bandit scheme [5], and even reinforcement learning ones [28], [32]. Remark 2. The exploration-exploitation step for taking actions guarantees continuously visiting state-action pairs, and ensures convergence to the optimal Q-function [23]. Instead of thegreedy exploration-exploitation step in Alg. 1, one can employ the upper confidence bound scheme [2]. Technically, any exploration-exploitation scheme should be greedy in the limit of infinite exploration (GLIE) [27, p. 840]. An example obeying GLIE is the -greedy algorithm [27, p. 840] with τ = 1/τ . It converges to an optimal policy, albeit at a very slow rate. On the other hand, using a constant τ = approaches the optimal Q * (·, ·) faster, but its exact convergence is not guaranteed as it is not GLIE.
Clearly, finding Q π * (s s s, a a a) entails estimating a function defined over state and action spaces. In several applications however, at least one of the two vector variables is either continuous or takes values from an alphabet of high cardinality. Revisiting every state-action pair in such settings is impossible, due to the so-called curse of dimensionality -a typical case in practice. To deal with it function approximation techniques offer as a promising solution [33]. These aim at finding the original Q-function over all feasible state-action pairs, by judicious generalization from a few observed pairs.
Early function approximators for reinforcement learning design good hand-crafted features that can properly approximate V π * (·), Q π * (·, ·), or, π * (·) [8]. Their applicability has only been limited to domains, where such features can be discovered, or, to state spaces that are low dimensional [11].
Deep learning approaches on the other hand, have recently demonstrated remarkable potential in applications such as object detection, speech recognition, and language translation, to name a few [1]. This is because DNNs are capable of extracting compact low-dimensional features from high-dimensional data. Wedding DNNs with reinforcement learning results in 'deep reinforcement learning' that can effectively deal with the curse of dimensionality, by eliminating the need of handcrafting good features. These considerations have inspired the use of DNNs to estimate either the Q-function, value function, or, the policy [1]. The most remarkable success in playing AI games adopted DNNs to estimate the Q-function, what is termed deep Q-network (DQN) in [21].
Prompted by this success, the next leverages the power of function approximation through DNNs to develop an adaptive caching scheme for the parent node.

V. ADAPTIVE DQN-BASED CACHING
To find the optimal caching policy π * for the parent node, the success of function approximators for RL (e.g., [8]), motivated us to pursue a parametric approach to estimating Q(s s s, a a a) with a DNN (cf. (13)). This DNN has as input pairs of vectors (s s s, a a a), and as scalar output the corresponding estimated Q (s s s, a a a) values. Clearly, the joint state-action space of the sought Q-function has cardinality |A×S| = |A||S|. To reduce the search space, we shall take a parametric DQN approach [21] that we adapt to our setup, as elaborated next.
Consider a deep feedforward NN with L fully connected layers, with input the F × 1 state vector s s s(τ ) as in (3), and F × 1 output cost vector o o o(τ + 1) [21]. Note that input does not include the action vector a a a. Each hidden layer l ∈ {2, . . . , L − 1} comprises n l neurons with rectified  Fig. 5 to obtain the action vector a a a(τ + 1). We can think of our DQN as a 'soft policy' function estimator, and o o o(τ + 1) as a 'predicted cost' or a 'soft action' vector for interval τ + 1, whose 'hard thresholding' yields a a a(τ + 1). It will turn out that excluding a a a from the DQN input and picking it up at the output lowers the search space from |A||S| to |S|.
Since the dimensionality of θ θ θ can be much smaller than |S||A|, the DQN is efficiently trained with few experiences, and generalizes to unseen state vectors. Unfortunately, DQN model inaccuracy can propagate in the cost prediction error in (18) that can cause instability in (20), which can lead to performance degradation, and even divergence [31], [38]. Moreover, (20) leverages solely a single most recent experience E E E τ +1 . These limitations will be mitigated as elaborated next.

A. Target Network and Experience Replay
NN function approximation, along with the loss (19) and the update (20), often result in unstable RL algorithms [21], [19]. This is due to: i) correlated experiences used to update the DQN parameters θ θ θ; and, ii) the influence of any change in policy on subsequent experiences and vice versa.
Possible remedies include the so-called experience replay and target network to update the DQN weights. In experience replay, the parent node stores all past experiences E E E τ in E := {E E E 1 , . . . , E E E τ }, and utilizes a batch of B uniformly sampled experiences from this data set, namely . By sampling and replaying previously observed experiences, experience replay can overcome the two challenges. On the other hand, to obtain decorrelated target values in (18), a second NN (called target network) with structure identical to the DQN is invoked with parameter vector θ Tar . Interestingly, θ Tar can be periodically replaced with θ τ every C training iterations of the DQN, which enables the target network to smooth out fluctuations in updating the DQN [21].
With a randomly sampled experience E E E iτ ∈ E, the prediction error with the target cost-to-go estimated using the target network (instead of the DQN) is (1 − a a a(i τ +1)).

21
Uniformly sample B experiences from E

22
Find ∇L Tar (θ) for these samples, using (22) 23 Update θ τ +1 = θ τ − β τ ∇L Tar (θ) 24 If mod(τ, C) = 0, then update θ θ θ Tar = θ θ θ τ 25 end (18) is found by using the most recent experience, while the experience here is randomly drawn from past experiences in E. As a result, the loss function becomes L Tar (θ θ θ) = E δ δ δ Tar (θ θ θ; E E E) 2 2 (22) where the expectation is taken with respect to the uniformly sampled experience E E E. In practice however, only a batch of B samples is available and used to update θ θ θ τ , so the expectation will be replaced with the sample mean. Finally, following a gradient descent step over the sampled experiences, we have Both the experience replay and the target network help stabilize the DQN updates. Incorporating these remedies, Alg. 2 tabulates our deep reinforcement learning based adaptive caching scheme for the parent node.

VI. NUMERICAL TESTS
In this section, we present several numerical tests to assess the performance of our deep reinforcement learning based caching schemes that rely on Algs. 1 and 2.
The first experiment considers a simple setup, consisting of a parent node that directly serves user file requests. A total of F = 50 files were assumed in the cloud, each having the same size, while M 0 = 5 files can be stored in the cache, amounting to 10% of the total. The file popularities were drawn randomly from [0, 1], invariant across all simulated time instants, but unknown to the caching entity. A fully connected feed-forward NN of 3 layers was implemented for DQN with each layer comprising 50 hidden neurons. For simplicity, we assume that the dataset E can store a number R of most recent experiences, which is also known as the replay memory. It was set to R = 10 in our test, along with mini-batch size B = 1 to update the target network every C = 10 iterations. Fig. 6 shows the convergence of the DQN parameter θ θ θ τ to that of the target network θ θ θ Tar . Since θ θ θ Tar is updated with θ θ θ τ per C iteration, the error ||θ τ − θ Tar || 2 vanishes periodically. While the error changes just in a few iterations after the θ θ θ Tar update, it is constant in several iterations before the next θ θ θ Tar update, suggesting that θ θ θ τ is fixed across those iterations. This motivates investigating the impact of C on θ θ θ τ 's convergence. Fig. 7 shows the convergence of θ θ θ τ to θ θ θ Tar for C = 20, 5, 3, 2. Starting with C = 20, vector θ θ θ τ is not updated in most iterations between two consecutive updates of θ θ θ Tar . Clearly, the smaller C is, the faster the convergence for θ θ θ τ is achieved. Indeed, this is a direct result of having static file popularity throughout the simulated intervals. As will be shown later, having small C values is also preferable in dynamic settings.  For the second experiment, a similar setup having solely the parent node, plus M 0 = 5 and F = 50, was considered. Dynamic file popularities were generated with time evolutions modeled by Markov random processes, and again they are unknown to the caching node. An illustration of temporal evolution of such file popularities is depicted in Fig. 8. Here, each color represents the number of received requests for a particular file. Convergence of θ τ to θ Tar with C = 10, is corroborated in Fig. 9. Compared with Figs. 6 and 7, the error here never vanishes, which implies that the DQN's and the target network's parameters are continuously changing. Intuitively, this is due to the dynamic evolution of file requests. As will be shown in the sequel, although θ τ never converges, it is powerful enough to find the proper files to store. In this regard, we define a non-causal optimal policy as benchmark, which unrealistically assumes knowledge of future requests and stores the most frequently requested files. In fact, this is the best policy that one may ideally implement. By drawing 100 samples randomly from all 2, 000 time intervals, the actual cost reduction obtained by adopting this optimal policy is plotted with a red line in Fig. 10. Under the same setting, the cost reduction achieved by following our proposed deep reinforcement learning based caching approach is plotted in blue. This figure shows remarkable performance of our proposed approach relative to the non-causal optimal one. Furthermore, the empirical CDFs for reduced costs obtained over these 100 random samples, are presented in Fig. 11. This further verifies how our deep reinforcement learning based approach performs relatives to the optimal policy. Finally, Fig. 12 shows convergence of the DQN parameters to the target network's parameters when C = 20, 10, 5, 2. Again, the error never goes to zero. Further, similar to the static case (cf. Fig. 7), smaller C values typically yield faster error decay.
To further examine performance of the proposed approach, consider now N = 20 leaf nodes are connected to the parent node. They receive user file requests during fast time slots, where there are T = 2 fast time slots within every slow timescale interval. File requests exhibit different Markovian evolution locally at different leaf nodes. There are a total of 2, 000 time intervals, a total of F = 50 files, M 0 = 5 for the parent node, and the weights are all set to w n = 1, ∀n.
Consider first the leaf nodes have no caching capacity. Fig. 13 compares the deep reinforcement learning based caching policy with baselines in terms of the empirical CDF of reduced cost. Here, the random policy, namely storing files randomly, and the worst case is a policy which stores the least popular files while assumed to have non-causal information about file requests. Corresponding instantaneous reduced costs, at the sampled intervals, are reported in Fig. 14. Both Figs. 13 and 14, clearly show the remarkable performance of the deep reinforcement learning based policy relative to the optimal one. In addition, compared with previous results, it becomes clear that, the more the leaf nodes, the better the performance of our proposed approach.
Finally, consider every leaf node has a caching capacity of storing M n = 5 files, and it implements a local caching policy π n . But these local caching policies are unknown to the parent node. To find the optimal caching decision, the parent node not only learns the file popularities and their temporal evolution, but also should learn its leaf nodes' caching policies. Figs. 15 and 16 compares the deep reinforcement learning based policy with the benchmarks. Again, these results cor-  roborate the remarkable performance of our proposed deep reinforcement learning caching based scheme. Compared with previous results, the slight degradation of deep reinforcement learning performance, is a consequence of considering local caching policies at the leaf nodes. Nevertheless, given the results in Fig. 16, the deep reinforcement learning based policy performs remarkably well compared with the optimal policy.

VII. CONCLUSIONS
Caching highly popular contents across distributed caching entities can substantially reduce the heavy burden of content delivery in modern data networks. This paper considered a network caching setup, comprising a parent node connected to several leaf nodes to serve end user file requests. Since the leaf nodes are closer to end users, they observe file requests in a fast timescale, and could thus serve requests, either through their locally stored files or by fetching files from the parent node. Observing aggregate requests from all leaf nodes, the parent node makes caching decisions in a slowertime scale. This two-time scale caching task was cast in this paper using a reinforcement learning approach. An efficient caching policy leveraging deep reinforcement learning was introduced, and shown capable of learning-and-adapting to the This work also opens up several directions for future research, including multi-armed bandit online learning [18], and distributed deep reinforcement learning using recurrent NNs [22] for future spatio-temporal file request prediction.