Deep Bayesian Bandits: Exploring in Online Personalized Recommendations
  Recommender systems trained in a continuous learning fashion are plagued by
the feedback loop problem, also known as algorithmic bias. This causes a newly
trained model to act greedily and favor items that have already been engaged by
users. This behavior is particularly harmful in personalised ads
recommendations, as it can also cause new campaigns to remain unexplored.
Exploration aims to address this limitation by providing new information about
the environment, which encompasses user preference, and can lead to higher
long-term reward. In this work, we formulate a display advertising recommender
as a contextual bandit and implement exploration techniques that require
sampling from the posterior distribution of click-through-rates in a
computationally tractable manner. Traditional large-scale deep learning models
do not provide uncertainty estimates by default. We approximate these
uncertainty measurements of the predictions by employing a bootstrapped model
with multiple heads and dropout units. We benchmark a number of different
models in an offline simulation environment using a publicly available dataset
of user-ads engagements. We test our proposed deep Bayesian bandits algorithm
in the offline simulation and online AB setting with large-scale production
traffic, where we demonstrate a positive gain of our exploration model.
