Collapsing Bandits and Their Application to Public Health Interventions
  We propose and study Collpasing Bandits, a new restless multi-armed bandit
(RMAB) setting in which each arm follows a binary-state Markovian process with
a special structure: when an arm is played, the state is fully observed, thus
"collapsing" any uncertainty, but when an arm is passive, no observation is
made, thus allowing uncertainty to evolve. The goal is to keep as many arms in
the "good" state as possible by planning a limited budget of actions per round.
Such Collapsing Bandits are natural models for many healthcare domains in which
workers must simultaneously monitor patients and deliver interventions in a way
that maximizes the health of their patient cohort. Our main contributions are
as follows: (i) Building on the Whittle index technique for RMABs, we derive
conditions under which the Collapsing Bandits problem is indexable. Our
derivation hinges on novel conditions that characterize when the optimal
policies may take the form of either "forward" or "reverse" threshold policies.
(ii) We exploit the optimality of threshold policies to build fast algorithms
for computing the Whittle index, including a closed-form. (iii) We evaluate our
algorithm on several data distributions including data from a real-world
healthcare task in which a worker must monitor and deliver interventions to
maximize their patients' adherence to tuberculosis medication. Our algorithm
achieves a 3-order-of-magnitude speedup compared to state-of-the-art RMAB
techniques while achieving similar performance.
