EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits
  In this paper, we propose a novel neural exploration strategy in contextual
bandits, EE-Net, distinct from the standard UCB-based and TS-based approaches.
Contextual multi-armed bandits have been studied for decades with various
applications. To solve the exploitation-exploration tradeoff in bandits, there
are three main techniques: epsilon-greedy, Thompson Sampling (TS), and Upper
Confidence Bound (UCB). In recent literature, linear contextual bandits have
adopted ridge regression to estimate the reward function and combine it with TS
or UCB strategies for exploration. However, this line of works explicitly
assumes the reward is based on a linear function of arm vectors, which may not
be true in real-world datasets. To overcome this challenge, a series of neural
bandit algorithms have been proposed, where a neural network is used to learn
the underlying reward function and TS or UCB are adapted for exploration.
Instead of calculating a large-deviation based statistical bound for
exploration like previous methods, we propose "EE-Net", a novel neural-based
exploration strategy. In addition to using a neural network (Exploitation
network) to learn the reward function, EE-Net uses another neural network
(Exploration network) to adaptively learn potential gains compared to the
currently estimated reward for exploration. Then, a decision-maker is
constructed to combine the outputs from the Exploitation and Exploration
networks. We prove that EE-Net can achieve $\mathcal{O}(\sqrt{T\log T})$ regret
and show that EE-Net outperforms existing linear and neural contextual bandit
baselines on real-world datasets.
