MarlRank: Multi-agent Reinforced Learning to Rank
  When estimating the relevancy between a query and a document, ranking models
largely neglect the mutual information among documents. A common wisdom is that
if two documents are similar in terms of the same query, they are more likely
to have similar relevance score. To mitigate this problem, in this paper, we
propose a multi-agent reinforced ranking model, named MarlRank. In particular,
by considering each document as an agent, we formulate the ranking process as a
multi-agent Markov Decision Process (MDP), where the mutual interactions among
documents are incorporated in the ranking process. To compute the ranking list,
each document predicts its relevance to a query considering not only its own
query-document features but also its similar documents features and actions. By
defining reward as a function of NDCG, we can optimize our model directly on
the ranking performance measure. Our experimental results on two LETOR
benchmark datasets show that our model has significant performance gains over
the state-of-art baselines. We also find that the NDCG shows an overall
increasing trend along with the step of interactions, which demonstrates that
the mutual information among documents helps improve the ranking performance.
