Metareasoning in Modular Software Systems: On-the-Fly Configuration
  using Reinforcement Learning with Rich Contextual Representations
  Assemblies of modular subsystems are being pressed into service to perform
sensing, reasoning, and decision making in high-stakes, time-critical tasks in
such areas as transportation, healthcare, and industrial automation. We address
the opportunity to maximize the utility of an overall computing system by
employing reinforcement learning to guide the configuration of the set of
interacting modules that comprise the system. The challenge of doing
system-wide optimization is a combinatorial problem. Local attempts to boost
the performance of a specific module by modifying its configuration often leads
to losses in overall utility of the system's performance as the distribution of
inputs to downstream modules changes drastically. We present metareasoning
techniques which consider a rich representation of the input, monitor the state
of the entire pipeline, and adjust the configuration of modules on-the-fly so
as to maximize the utility of a system's operation. We show significant
improvement in both real-world and synthetic pipelines across a variety of
reinforcement learning techniques.
