Reinforcement Learning with a Disentangled Universal Value Function for
  Item Recommendation
  In recent years, there are great interests as well as challenges in applying
reinforcement learning (RL) to recommendation systems (RS). In this paper, we
summarize three key practical challenges of large-scale RL-based recommender
systems: massive state and action spaces, high-variance environment, and the
unspecific reward setting in recommendation. All these problems remain largely
unexplored in the existing literature and make the application of RL
challenging. We develop a model-based reinforcement learning framework, called
GoalRec. Inspired by the ideas of world model (model-based), value function
estimation (model-free), and goal-based RL, a novel disentangled universal
value function designed for item recommendation is proposed. It can generalize
to various goals that the recommender may have, and disentangle the stochastic
environmental dynamics and high-variance reward signals accordingly. As a part
of the value function, free from the sparse and high-variance reward signals, a
high-capacity reward-independent world model is trained to simulate complex
environmental dynamics under a certain goal. Based on the predicted
environmental dynamics, the disentangled universal value function is related to
the user's future trajectory instead of a monolithic state and a scalar reward.
We demonstrate the superiority of GoalRec over previous approaches in terms of
the above three practical challenges in a series of simulations and a real
application.
