Knapsack based Optimal Policies for Budget-Limited Multi-Armed Bandits
  In budget-limited multi-armed bandit (MAB) problems, the learner's actions
are costly and constrained by a fixed budget. Consequently, an optimal
exploitation policy may not be to pull the optimal arm repeatedly, as is the
case in other variants of MAB, but rather to pull the sequence of different
arms that maximises the agent's total reward within the budget. This difference
from existing MABs means that new approaches to maximising the total reward are
required. Given this, we develop two pulling policies, namely: (i) KUBE; and
(ii) fractional KUBE. Whereas the former provides better performance up to 40%
in our experimental settings, the latter is computationally less expensive. We
also prove logarithmic upper bounds for the regret of both policies, and show
that these bounds are asymptotically optimal (i.e. they only differ from the
best possible regret by a constant factor).
