Supervised Advantage Actor-Critic for Recommender Systems
  Casting session-based or sequential recommendation as reinforcement learning
(RL) through reward signals is a promising research direction towards
recommender systems (RS) that maximize cumulative profits. However, the direct
use of RL algorithms in the RS setting is impractical due to challenges like
off-policy training, huge action spaces and lack of sufficient reward signals.
Recent RL approaches for RS attempt to tackle these challenges by combining RL
and (self-)supervised sequential learning, but still suffer from certain
limitations. For example, the estimation of Q-values tends to be biased toward
positive values due to the lack of negative reward signals. Moreover, the
Q-values also depend heavily on the specific timestamp of a sequence.
  To address the above problems, we propose negative sampling strategy for
training the RL component and combine it with supervised sequential learning.
We call this method Supervised Negative Q-learning (SNQN). Based on sampled
(negative) actions (items), we can calculate the "advantage" of a positive
action over the average case, which can be further utilized as a normalized
weight for learning the supervised sequential part. This leads to another
learning framework: Supervised Advantage Actor-Critic (SA2C). We instantiate
SNQN and SA2C with four state-of-the-art sequential recommendation models and
conduct experiments on two real-world datasets. Experimental results show that
the proposed approaches achieve significantly better performance than
state-of-the-art supervised methods and existing self-supervised RL methods .
Code will be open-sourced.
