Offline Planning and Online Learning under Recovering Rewards
  Motivated by emerging applications such as live-streaming e-commerce,
promotions and recommendations, we introduce and solve a general class of
non-stationary multi-armed bandit problems that have the following two
features: (i) the decision maker can pull and collect rewards from up to
$K\,(\ge 1)$ out of $N$ different arms in each time period; (ii) the expected
reward of an arm immediately drops after it is pulled, and then
non-parametrically recovers as the arm's idle time increases. With the
objective of maximizing the expected cumulative reward over $T$ time periods,
we design a class of ``Purely Periodic Policies'' that jointly set a period to
pull each arm. For the proposed policies, we prove performance guarantees for
both the offline problem and the online problems. For the offline problem when
all model parameters are known, the proposed periodic policy obtains an
approximation ratio that is at the order of $1-\mathcal O(1/\sqrt{K})$, which
is asymptotically optimal when $K$ grows to infinity. For the online problem
when the model parameters are unknown and need to be dynamically learned, we
integrate the offline periodic policy with the upper confidence bound procedure
to construct on online policy. The proposed online policy is proved to
approximately have $\widetilde{\mathcal O}(N\sqrt{T})$ regret against the
offline benchmark. Our framework and policy design may shed light on broader
offline planning and online learning applications with non-stationary and
recovering rewards.
