An efficient algorithm for contextual bandits with knapsacks, and an
  extension to concave objectives
  We consider a contextual version of multi-armed bandit problem with global
knapsack constraints. In each round, the outcome of pulling an arm is a scalar
reward and a resource consumption vector, both dependent on the context, and
the global knapsack constraints require the total consumption for each resource
to be below some pre-fixed budget. The learning agent competes with an
arbitrary set of context-dependent policies. This problem was introduced by
Badanidiyuru et al. (2014), who gave a computationally inefficient algorithm
with near-optimal regret bounds for it. We give a computationally efficient
algorithm for this problem with slightly better regret bounds, by generalizing
the approach of Agarwal et al. (2014) for the non-constrained version of the
problem. The computational time of our algorithm scales logarithmically in the
size of the policy space. This answers the main open question of Badanidiyuru
et al. (2014). We also extend our results to a variant where there are no
knapsack constraints but the objective is an arbitrary Lipschitz concave
function of the sum of outcome vectors.
