RLCache: Automated Cache Management Using Reinforcement Learning
  This study investigates the use of reinforcement learning to guide a general
purpose cache manager decisions. Cache managers directly impact the overall
performance of computer systems. They govern decisions about which objects
should be cached, the duration they should be cached for, and decides on which
objects to evict from the cache if it is full. These three decisions impact
both the cache hit rate and size of the storage that is needed to achieve that
cache hit rate. An optimal cache manager will avoid unnecessary operations,
maximise the cache hit rate which results in fewer round trips to a slower
backend storage system, and minimise the size of storage needed to achieve a
high hit-rate.
  This project investigates using reinforcement learning in cache management by
designing three separate agents for each of the cache manager tasks.
Furthermore, the project investigates two advanced reinforcement learning
architectures for multi-decision problems: a single multi-task agent and a
multi-agent. We also introduce a framework to simplify the modelling of
computer systems problems as a reinforcement learning task. The framework
abstracts delayed experiences observations and reward assignment in computer
systems while providing a flexible way to scale to multiple agents.
  Simulation results based on an established database benchmark system show
that reinforcement learning agents can achieve a higher cache hit rate over
heuristic driven algorithms while minimising the needed space. They are also
able to adapt to a changing workload and dynamically adjust their caching
strategy accordingly. The proposed cache manager model is generic and
applicable to other types of caches, such as file system caches. This project
is the first, to our knowledge, to model cache manager decisions as a
multi-task control problem.
