Robust Restless Bandits: Tackling Interval Uncertainty with Deep
  Reinforcement Learning
  We introduce Robust Restless Bandits, a challenging generalization of
restless multi-arm bandits (RMAB). RMABs have been widely studied for
intervention planning with limited resources. However, most works make the
unrealistic assumption that the transition dynamics are known perfectly,
restricting the applicability of existing methods to real-world scenarios. To
make RMABs more useful in settings with uncertain dynamics: (i) We introduce
the Robust RMAB problem and develop solutions for a minimax regret objective
when transitions are given by interval uncertainties; (ii) We develop a double
oracle algorithm for solving Robust RMABs and demonstrate its effectiveness on
three experimental domains; (iii) To enable our double oracle approach, we
introduce RMABPPO, a novel deep reinforcement learning algorithm for solving
RMABs. RMABPPO hinges on learning an auxiliary "$\lambda$-network" that allows
each arm's learning to decouple, greatly reducing sample complexity required
for training; (iv) Under minimax regret, the adversary in the double oracle
approach is notoriously difficult to implement due to non-stationarity. To
address this, we formulate the adversary oracle as a multi-agent reinforcement
learning problem and solve it with a multi-agent extension of RMABPPO, which
may be of independent interest as the first known algorithm for this setting.
Code is available at https://github.com/killian-34/RobustRMAB.
