A Change-Detection Based Thompson Sampling Framework for Non-Stationary
  Bandits
  We consider a non-stationary two-armed bandit framework and propose a
change-detection based Thompson sampling (TS) algorithm, named TS with
change-detection (TS-CD), to keep track of the dynamic environment. The
non-stationarity is modeled using a Poisson arrival process, which changes the
mean of the rewards on each arrival. The proposed strategy compares the
empirical mean of the recent rewards of an arm with the estimate of the mean of
the rewards from its history. It detects a change when the empirical mean
deviates from the mean estimate by a value larger than a threshold. Then, we
characterize the lower bound on the duration of the time-window for which the
bandit framework must remain stationary for TS-CD to successfully detect a
change when it occurs. Consequently, our results highlight an upper bound on
the parameter for the Poisson arrival process, for which the TS-CD achieves
asymptotic regret optimality with high probability. Finally, we validate the
efficacy of TS-CD by testing it for edge-control of radio access technique
(RAT)-selection in a wireless network. Our results show that TS-CD not only
outperforms the classical max-power RAT selection strategy but also other
actively adaptive and passively adaptive bandit algorithms that are designed
for non-stationary environments.
