Cross DQN: Cross Deep Q Network for Ads Allocation in Feed
  E-commerce platforms usually display a mixed list of ads and organic items in
feed. One key problem is to allocate the limited slots in the feed to maximize
the overall revenue as well as improve user experience, which requires a good
model for user preference. Instead of modeling the influence of individual
items on user behaviors, the arrangement signal models the influence of the
arrangement of items and may lead to a better allocation strategy. However,
most of previous strategies fail to model such a signal and therefore result in
suboptimal performance. In addition, the percentage of ads exposed (PAE) is an
important indicator in ads allocation. Excessive PAE hurts user experience
while too low PAE reduces platform revenue. Therefore, how to constrain the PAE
within a certain range while keeping personalized recommendation under the PAE
constraint is a challenge. In this paper, we propose Cross Deep Q Network
(Cross DQN) to extract the crucial arrangement signal by crossing the
embeddings of different items and modeling the crossed sequence by
multi-channel attention. Besides, we propose an auxiliary loss for batch-level
constraint on PAE to tackle the above-mentioned challenge. Our model results in
higher revenue and better user experience than state-of-the-art baselines in
offline experiments. Moreover, our model demonstrates a significant improvement
in the online A/B test and has been fully deployed on Meituan feed to serve
more than 300 millions of customers.
