Improving Long-Term Metrics in Recommendation Systems using
  Short-Horizon Reinforcement Learning
  We study session-based recommendation scenarios where we want to recommend
items to users during sequential interactions to improve their long-term
utility. Optimizing a long-term metric is challenging because the learning
signal (whether the recommendations achieved their desired goals) is delayed
and confounded by other user interactions with the system. Targeting
immediately measurable proxies such as clicks can lead to suboptimal
recommendations due to misalignment with the long-term metric. We develop a new
reinforcement learning algorithm called Short Horizon Policy Improvement (SHPI)
that approximates policy-induced drift in user behavior across sessions. SHPI
is a straightforward modification of episodic RL algorithms for session-based
recommendation, that additionally gives an appropriate termination bonus in
each session. Empirical results on four recommendation tasks show that SHPI can
outperform state-of-the-art recommendation techniques like matrix factorization
with offline proxy signals, bandits with myopic online proxies, and RL
baselines with limited amounts of user interaction.
