Constrained Restless Bandits for Dynamic Scheduling in Cyber-Physical
  Systems
  This paper studies a class of constrained restless multi-armed bandits
(CRMAB). The constraints are in the form of time varying set of actions (set of
available arms). This variation can be either stochastic or semi-deterministic.
Given a set of arms, a fixed number of them can be chosen to be played in each
decision interval. The play of each arm yields a state dependent reward. The
current states of arms are partially observable through binary feedback signals
from arms that are played. The current availability of arms is fully
observable. The objective is to maximize long term cumulative reward. The
uncertainty about future availability of arms along with partial state
information makes this objective challenging. Applications for CRMAB can be
found in resource allocation in cyber-physical systems involving components
with time varying availability.
  First, this optimization problem is analyzed using Whittle's index policy. To
this end, a constrained restless single-armed bandit is studied. It is shown to
admit a threshold-type optimal policy and is also indexable. An algorithm to
compute Whittle's index is presented. An alternate solution method with lower
complexity is also presented in the form of an online rollout policy. A
detailed discussion on the complexity of both these schemes is also presented,
which suggests that online rollout policy with short look ahead is simpler to
implement than Whittle's index computation. Further, upper bounds on the value
function are derived in order to estimate the degree of sub-optimality of
various solutions. The simulation study compares the performance of Whittle's
index, online rollout, myopic and modified Whittle's index policies.
