Efficient Algorithms for Finite Horizon and Streaming Restless
  Multi-Armed Bandit Problems
  We propose Streaming Bandits, a Restless Multi Armed Bandit (RMAB) framework
in which heterogeneous arms may arrive and leave the system after staying on
for a finite lifetime. Streaming Bandits naturally capture the health
intervention planning problem, where health workers must manage the health
outcomes of a patient cohort while new patients join and existing patients
leave the cohort each day. Our contributions are as follows: (1) We derive
conditions under which our problem satisfies indexability, a precondition that
guarantees the existence and asymptotic optimality of the Whittle Index
solution for RMABs. We establish the conditions using a polytime reduction of
the Streaming Bandit setup to regular RMABs. (2) We further prove a phenomenon
that we call index decay, whereby the Whittle index values are low for short
residual lifetimes driving the intuition underpinning our algorithm. (3) We
propose a novel and efficient algorithm to compute the index-based solution for
Streaming Bandits. Unlike previous methods, our algorithm does not rely on
solving the costly finite horizon problem on each arm of the RMAB, thereby
lowering the computational complexity compared to existing methods. (4)
Finally, we evaluate our approach via simulations run on realworld data sets
from a tuberculosis patient monitoring task and an intervention planning task
for improving maternal healthcare, in addition to other synthetic domains.
Across the board, our algorithm achieves a 2-orders-of-magnitude speed-up over
existing methods while maintaining the same solution quality.
