Regression Oracles and Exploration Strategies for Short-Horizon
  Multi-Armed Bandits
  This paper explores multi-armed bandit (MAB) strategies in very short horizon
scenarios, i.e., when the bandit strategy is only allowed very few interactions
with the environment. This is an understudied setting in the MAB literature
with many applications in the context of games, such as player modeling.
Specifically, we pursue three different ideas. First, we explore the use of
regression oracles, which replace the simple average used in strategies such as
epsilon-greedy with linear regression models. Second, we examine different
exploration patterns such as forced exploration phases. Finally, we introduce a
new variant of the UCB1 strategy called UCBT that has interesting properties
and no tunable parameters. We present experimental results in a domain
motivated by exergames, where the goal is to maximize a player's daily steps.
Our results show that the combination of epsilon-greedy or epsilon-decreasing
with regression oracles outperforms all other tested strategies in the short
horizon setting.
