To update or not to update? Delayed Nonparametric Bandits with
  Randomized Allocation
  Delayed rewards problem in contextual bandits has been of interest in various
practical settings. We study randomized allocation strategies and provide an
understanding on how the exploration-exploitation tradeoff is affected by
delays in observing the rewards. In randomized strategies, the extent of
exploration-exploitation is controlled by a user-determined exploration
probability sequence. In the presence of delayed rewards, one may choose
between using the original exploration sequence that updates at every time
point or update the sequence only when a new reward is observed, leading to two
competing strategies. In this work, we show that while both strategies may lead
to strong consistency in allocation, the property holds for a wider scope of
situations for the latter. However, for finite sample performance, we
illustrate that both strategies have their own advantages and disadvantages,
depending on the severity of the delay and underlying reward generating
mechanisms.
