Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in
  Application to Preventive Healthcare
  In many public health settings, it is important for patients to adhere to
health programs, such as taking medications and periodic health checks.
Unfortunately, beneficiaries may gradually disengage from such programs, which
is detrimental to their health. A concrete example of gradual disengagement has
been observed by an organization that carries out a free automated call-based
program for spreading preventive care information among pregnant women. Many
women stop picking up calls after being enrolled for a few months. To avoid
such disengagements, it is important to provide timely interventions. Such
interventions are often expensive and can be provided to only a small fraction
of the beneficiaries. We model this scenario as a restless multi-armed bandit
(RMAB) problem, where each beneficiary is assumed to transition from one state
to another depending on the intervention. Moreover, since the transition
probabilities are unknown a priori, we propose a Whittle index based Q-Learning
mechanism and show that it converges to the optimal solution. Our method
improves over existing learning-based methods for RMABs on multiple benchmarks
from literature and also on the maternal healthcare dataset.
