Sequential Search with Off-Policy Reinforcement Learning
  Recent years have seen a significant amount of interests in Sequential
Recommendation (SR), which aims to understand and model the sequential user
behaviors and the interactions between users and items over time. Surprisingly,
despite the huge success Sequential Recommendation has achieved, there is
little study on Sequential Search (SS), a twin learning task that takes into
account a user's current and past search queries, in addition to behavior on
historical query sessions. The SS learning task is even more important than the
counterpart SR task for most of E-commence companies due to its much larger
online serving demands as well as traffic volume.
  To this end, we propose a highly scalable hybrid learning model that consists
of an RNN learning framework leveraging all features in short-term user-item
interactions, and an attention model utilizing selected item-only features from
long-term interactions. As a novel optimization step, we fit multiple short
user sequences in a single RNN pass within a training batch, by solving a
greedy knapsack problem on the fly. Moreover, we explore the use of off-policy
reinforcement learning in multi-session personalized search ranking.
Specifically, we design a pairwise Deep Deterministic Policy Gradient model
that efficiently captures users' long term reward in terms of pairwise
classification error. Extensive ablation experiments demonstrate significant
improvement each component brings to its state-of-the-art baseline, on a
variety of offline and online metrics.
