Bandits with Temporal Stochastic Constraints
  We study the effect of impairment on stochastic multi-armed bandits and
develop new ways to mitigate it. Impairment effect is the phenomena where an
agent only accrues reward for an action if they have played it at least a few
times in the recent past. It is practically motivated by repetition and recency
effects in domains such as advertising (here consumer behavior may require
repeat actions by advertisers) and vocational training (here actions are
complex skills that can only be mastered with repetition to get a payoff).
Impairment can be naturally modelled as a temporal constraint on the strategy
space, and we provide two novel algorithms that achieve sublinear regret, each
working with different assumptions on the impairment effect. We introduce a new
notion called bucketing in our algorithm design, and show how it can
effectively address impairment as well as a broader class of temporal
constraints. Our regret bounds explicitly capture the cost of impairment and
show that it scales (sub-)linearly with the degree of impairment. Our work
complements recent work on modeling delays and corruptions, and we provide
experimental evidence supporting our claims.
