Offline Reinforcement Learning for Mobile Notifications
  Mobile notification systems have taken a major role in driving and
maintaining user engagement for online platforms. They are interesting
recommender systems to machine learning practitioners with more sequential and
long-term feedback considerations. Most machine learning applications in
notification systems are built around response-prediction models, trying to
attribute both short-term impact and long-term impact to a notification
decision. However, a user's experience depends on a sequence of notifications
and attributing impact to a single notification is not always accurate, if not
impossible. In this paper, we argue that reinforcement learning is a better
framework for notification systems in terms of performance and iteration speed.
We propose an offline reinforcement learning framework to optimize sequential
notification decisions for driving user engagement. We describe a
state-marginalized importance sampling policy evaluation approach, which can be
used to evaluate the policy offline and tune learning hyperparameters. Through
simulations that approximate the notifications ecosystem, we demonstrate the
performance and benefits of the offline evaluation approach as a part of the
reinforcement learning modeling approach. Finally, we collect data through
online exploration in the production system, train an offline Double Deep
Q-Network and launch a successful policy online. We also discuss the practical
considerations and results obtained by deploying these policies for a
large-scale recommendation system use-case.
