Generator and Critic: A Deep Reinforcement Learning Approach for Slate
  Re-ranking in E-commerce
  The slate re-ranking problem considers the mutual influences between items to
improve user satisfaction in e-commerce, compared with the point-wise ranking.
Previous works either directly rank items by an end to end model, or rank items
by a score function that trades-off the point-wise score and the diversity
between items. However, there are two main existing challenges that are not
well studied: (1) the evaluation of the slate is hard due to the complex mutual
influences between items of one slate; (2) even given the optimal evaluation,
searching the optimal slate is challenging as the action space is exponentially
large. In this paper, we present a novel Generator and Critic slate re-ranking
approach, where the Critic evaluates the slate and the Generator ranks the
items by the reinforcement learning approach. We propose a Full Slate Critic
(FSC) model that considers the real impressed items and avoids the impressed
bias of existing models. For the Generator, to tackle the problem of large
action space, we propose a new exploration reinforcement learning algorithm,
called PPO-Exploration. Experimental results show that the FSC model
significantly outperforms the state of the art slate evaluation methods, and
the PPO-Exploration algorithm outperforms the existing reinforcement learning
methods substantially. The Generator and Critic approach improves both the
slate efficiency(4% gmv and 5% number of orders) and diversity in live
experiments on one of the largest e-commerce websites in the world.
