Bandit Learning with Delayed Impact of Actions
  We consider a stochastic multi-armed bandit (MAB) problem with delayed impact
of actions. In our setting, actions taken in the past impact the arm rewards in
the subsequent future. This delayed impact of actions is prevalent in the real
world. For example, the capability to pay back a loan for people in a certain
social group might depend on historically how frequently that group has been
approved loan applications. If banks keep rejecting loan applications to people
in a disadvantaged group, it could create a feedback loop and further damage
the chance of getting loans for people in that group. In this paper, we
formulate this delayed and long-term impact of actions within the context of
multi-armed bandits. We generalize the bandit setting to encode the dependency
of this "bias" due to the action history during learning. The goal is to
maximize the collected utilities over time while taking into account the
dynamics created by the delayed impacts of historical actions. We propose an
algorithm that achieves a regret of $\tilde{\mathcal{O}}(KT^{2/3})$ and show a
matching regret lower bound of $\Omega(KT^{2/3})$, where $K$ is the number of
arms and $T$ is the learning horizon. Our results complement the bandit
literature by adding techniques to deal with actions with long-term impacts and
have implications in designing fair algorithms.
