When and Whom to Collaborate with in a Changing Environment: A
  Collaborative Dynamic Bandit Solution
  Collaborative bandit learning, i.e., bandit algorithms that utilize
collaborative filtering techniques to improve sample efficiency in online
interactive recommendation, has attracted much research attention as it enjoys
the best of both worlds. However, all existing collaborative bandit learning
solutions impose a stationary assumption about the environment, i.e., both user
preferences and the dependency among users are assumed static over time.
Unfortunately, this assumption hardly holds in practice due to users'
ever-changing interests and dependence relations, which inevitably costs a
recommender system sub-optimal performance in practice.
  In this work, we develop a collaborative dynamic bandit solution to handle a
changing environment for recommendation. We explicitly model the underlying
changes in both user preferences and their dependency relation as a stochastic
process. Individual user's preference is modeled by a mixture of globally
shared contextual bandit models with a Dirichlet Process prior. Collaboration
among users is thus achieved via Bayesian inference over the global bandit
models. Model selection and arm selection for each user are done via Thompson
sampling to balance exploitation and exploration. Our solution is proved to
maintain a standard $\tilde O(\sqrt{T})$ sublinear regret even in such a
challenging environment. And extensive empirical evaluations on both synthetic
and real-world datasets further confirmed the necessity of modeling a changing
environment and our algorithm's practical advantages against several
state-of-the-art online learning solutions.
