{'fraction_non_alphanumeric': 0.01972386587771203, 'fraction_numerical': 0.0019723865877712033, 'mean_word_length': 5.167173252279635, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Impact of Tokenization, Pretraining Task, and Transformer Depth on Text Ranking

This paper documents the University of Amsterdam’s participation in the TREC 2020 Deep Learning Track. Rather than motivated by engineering the best scoring system, our work is motivated by our interest in analysis, informing our understanding of the opportunities and challenges of transformers for text ranking. Speciﬁcally, we focus on the passage retrieval task where we try to answer three of sets of questions. First, transformers use diﬀerent tokenization than traditional IR approaches such as stemming and lemmatizing, leading to diﬀerent document representations. What is the eﬀect of modern preprocessing techniques on traditional retrieval algorithms? Our main observation is that the limited vocabulary of the BERT tokenizer is aﬀecting many long-tail tokens, which leads to large gains in eﬃciency at the cost of a small decrease in eﬀectiveness. Second, the eﬀectiveness of transformers is a result of the self-supervised pre-training task promoting general language understanding, ignorant of the speciﬁc demands of ranking tasks. Can we make further correlate queries and relevant passages in the pre-training task? Our main observation is that there is a whole continuum between the original self-supervised training task of BERT and the ﬁnal interaction ranker, and isolating ranking-aware pre-training tasks may leads to gains in eﬃciency (as these pretrained models can be reused for many tasks) as well as to gains in eﬀectiveness (in particular when limited data on the target task is available). Third, transformers combine large sequence length with many layers, with unclear what this deep semantics adds in the context of ranking. How complex do the models need to be in order to perform well on this task? Our main observation is that the deep layers of BERT lead to some, but relatively modest, gains in performance, but that the exact role of the presumed superior language understanding for search is far from clear.