Document Ranking with a Pretrained Sequence-to-Sequence Model
  This work proposes a novel adaptation of a pretrained sequence-to-sequence
model to the task of document ranking. Our approach is fundamentally different
from a commonly-adopted classification-based formulation of ranking, based on
encoder-only pretrained transformer architectures such as BERT. We show how a
sequence-to-sequence model can be trained to generate relevance labels as
"target words", and how the underlying logits of these target words can be
interpreted as relevance probabilities for ranking. On the popular MS MARCO
passage ranking task, experimental results show that our approach is at least
on par with previous classification-based models and can surpass them with
larger, more-recent models. On the test collection from the TREC 2004 Robust
Track, we demonstrate a zero-shot transfer-based approach that outperforms
previous state-of-the-art models requiring in-dataset cross-validation.
Furthermore, we find that our approach significantly outperforms an
encoder-only model in a data-poor regime (i.e., with few training examples). We
investigate this observation further by varying target words to probe the
model's use of latent knowledge.
