Trajectory Inspection: A Method for Iterative Clinician-Driven Design of
  Reinforcement Learning Studies
  Reinforcement learning (RL) has the potential to significantly improve
clinical decision making. However, treatment policies learned via RL from
observational data are sensitive to subtle choices in study design. We
highlight a simple approach, trajectory inspection, to bring clinicians into an
iterative design process for model-based RL studies. We identify where the
model recommends unexpectedly aggressive treatments or expects surprisingly
positive outcomes from its recommendations. Then, we examine clinical
trajectories simulated with the learned model and policy alongside the actual
hospital course. Applying this approach to recent work on RL for sepsis
management, we uncover a model bias towards discharge, a preference for high
vasopressor doses that may be linked to small sample sizes, and clinically
implausible expectations of discharge without weaning off vasopressors. We hope
that iterations of detecting and addressing the issues unearthed by our method
will result in RL policies that inspire more confidence in deployment.
