Bandit Learning with General Function Classes: Heteroscedastic Noise and
  Variance-dependent Regret Bounds
  We consider learning a stochastic bandit model, where the reward function
belongs to a general class of uniformly bounded functions, and the additive
noise can be heteroscedastic. Our model captures contextual linear bandits and
generalized linear bandits as special cases. While previous works (Kirschner
and Krause, 2018; Zhou et al., 2021) based on weighted ridge regression can
deal with linear bandits with heteroscedastic noise, they are not directly
applicable to our general model due to the curse of nonlinearity. In order to
tackle this problem, we propose a multi-level learning framework for the
general bandit model. The core idea of our framework is to partition the
observed data into different levels according to the variance of their
respective reward and perform online learning at each level collaboratively.
Under our framework, we first design an algorithm that constructs the
variance-aware confidence set based on empirical risk minimization and prove a
variance-dependent regret bound. For generalized linear bandits, we further
propose an algorithm based on follow-the-regularized-leader (FTRL) subroutine
and online-to-confidence-set conversion, which can achieve a tighter
variance-dependent regret under certain conditions.
