{'fraction_non_alphanumeric': 0.026917900403768506, 'fraction_numerical': 0.0026917900403768506, 'mean_word_length': 5.825688073394495, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Why a corpus-topics-relevance judgments framework isn’t enough: two simple retrieval challenges from the field

In this paper, I use two challenges to illustrate how retrieval tasks can fall outside the current corpus-topics-relevance judgment evaluation framework. The two challenges that span both desktop search and standard information retrieval are: (1) encountering new information or re-encountering forgotten information and (2) retrieval of the appropriate version of semi-redundant information, either from a personal information space or from a public store of datasets. These challenges probe the assumptions underlying corpus construction, topic selection, and relevance judgment by suggesting some common activities violate them.