Evaluating Reinforcement Learning Algorithms in Observational Health
  Settings
  Much attention has been devoted recently to the development of machine
learning algorithms with the goal of improving treatment policies in
healthcare. Reinforcement learning (RL) is a sub-field within machine learning
that is concerned with learning how to make sequences of decisions so as to
optimize long-term effects. Already, RL algorithms have been proposed to
identify decision-making strategies for mechanical ventilation, sepsis
management and treatment of schizophrenia. However, before implementing
treatment policies learned by black-box algorithms in high-stakes clinical
decision problems, special care must be taken in the evaluation of these
policies.
  In this document, our goal is to expose some of the subtleties associated
with evaluating RL algorithms in healthcare. We aim to provide a conceptual
starting point for clinical and computational researchers to ask the right
questions when designing and evaluating algorithms for new ways of treating
patients. In the following, we describe how choices about how to summarize a
history, variance of statistical estimators, and confounders in more ad-hoc
measures can result in unreliable, even misleading estimates of the quality of
a treatment policy. We also provide suggestions for mitigating these
effects---for while there is much promise for mining observational health data
to uncover better treatment policies, evaluation must be performed
thoughtfully.
