Optimizing AD Pruning of Sponsored Search with Reinforcement Learning
  Industrial sponsored search system (SSS) can be logically divided into three
modules: keywords matching, ad retrieving, and ranking. During ad retrieving,
the ad candidates grow exponentially. A query with high commercial value might
retrieve a great deal of ad candidates such that the ranking module could not
afford. Due to limited latency and computing resources, the candidates have to
be pruned earlier. Suppose we set a pruning line to cut SSS into two parts:
upstream and downstream. The problem we are going to address is: how to pick
out the best $K$ items from $N$ candidates provided by the upstream to maximize
the total system's revenue. Since the industrial downstream is very complicated
and updated quickly, a crucial restriction in this problem is that the
selection scheme should get adapted to the downstream. In this paper, we
propose a novel model-free reinforcement learning approach to fixing this
problem. Our approach considers downstream as a black-box environment, and the
agent sequentially selects items and finally feeds into the downstream, where
revenue would be estimated and used as a reward to improve the selection
policy. To the best of our knowledge, this is first time to consider the system
optimization from a downstream adaption view. It is also the first time to use
reinforcement learning techniques to tackle this problem. The idea has been
successfully realized in Baidu's sponsored search system, and online long time
A/B test shows remarkable improvements on revenue.
