Reinforcement Learning Based Cooperative Coded Caching under Dynamic
  Popularities in Ultra-Dense Networks
  For ultra-dense networks with wireless backhaul, caching strategy at small
base stations (SBSs), usually with limited storage, is critical to meet massive
high data rate requests. Since the content popularity profile varies with time
in an unknown way, we exploit reinforcement learning (RL) to design a
cooperative caching strategy with maximum-distance separable (MDS) coding. We
model the MDS coding based cooperative caching as a Markov decision process to
capture the popularity dynamics and maximize the long-term expected cumulative
traffic load served directly by the SBSs without accessing the macro base
station. For the formulated problem, we first find the optimal solution for a
small-scale system by embedding the cooperative MDS coding into Q-learning. To
cope with the large-scale case, we approximate the state-action value function
heuristically. The approximated function includes only a small number of
learnable parameters and enables us to propose a fast and efficient
action-selection approach, which dramatically reduces the complexity. Numerical
results verify the optimality/near-optimality of the proposed RL based
algorithms and show the superiority compared with the baseline schemes. They
also exhibit good robustness to different environments.
