Pythia: A Customizable Hardware Prefetching Framework Using Online
  Reinforcement Learning
  Past research has proposed numerous hardware prefetching techniques, most of
which rely on exploiting one specific type of program context information
(e.g., program counter, cacheline address) to predict future memory accesses.
These techniques either completely neglect a prefetcher's undesirable effects
(e.g., memory bandwidth usage) on the overall system, or incorporate
system-level feedback as an afterthought to a system-unaware prefetch
algorithm. We show that prior prefetchers often lose their performance benefit
over a wide range of workloads and system configurations due to their inherent
inability to take multiple different types of program context and system-level
feedback information into account while prefetching. In this paper, we make a
case for designing a holistic prefetch algorithm that learns to prefetch using
multiple different types of program context and system-level feedback
information inherent to its design.
  To this end, we propose Pythia, which formulates the prefetcher as a
reinforcement learning agent. For every demand request, Pythia observes
multiple different types of program context information to make a prefetch
decision. For every prefetch decision, Pythia receives a numerical reward that
evaluates prefetch quality under the current memory bandwidth usage. Pythia
uses this reward to reinforce the correlation between program context
information and prefetch decision to generate highly accurate, timely, and
system-aware prefetch requests in the future. Our extensive evaluations using
simulation and hardware synthesis show that Pythia outperforms multiple
state-of-the-art prefetchers over a wide range of workloads and system
configurations, while incurring only 1.03% area overhead over a desktop-class
processor and no software changes in workloads. The source code of Pythia can
be freely downloaded from https://github.com/CMU-SAFARI/Pythia.
