{'fraction_non_alphanumeric': 0.016591251885369532, 'fraction_numerical': 0.007541478129713424, 'mean_word_length': 4.734870317002882, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language Modelling in Information Retrieval

Information retrieval has been one of the backbones for the computer science and one of it main goals. As the new systems expand in size and complexity (having in mind an almost endless system such as Internet and search engines which index over milions of pages and sites) so does grow the importance of searching tasks. The main aim of the information retrieval is satisfying the information need of the user, i.e. traversing the given document collection (possibly indexed or preprocessed) and returning to the user some sort of presentation of important documents which are somehow connected or relevant to the user’s query. Over the past three decades, there has been a lot of research going on to solve these problems, but so far no method proved as the ultimate and the best. Although it seems that Google successfully solved most of the problems, the effectivness and details of its searching algorithm are still not public as well as the technology of the other searching engines. As late as 2003, the 40% accuracy limit was still not broken [1]. Language model approach to the information retrieval proved nevertheless as a very well-performing method in practice and allows us to model some of the special features of language that are omitted, ignored or tedious to model in other previous models. It is a quite new approach, concerning that the first pioneer research has taken place in the 1998 [8]. We will start with section 2 where I will try to describe briefly the methods which are preceding language model approach to information retrieval. Section 3 should try to explain some of the main differences and novelties of the approach compared with the older ones as well as similarities. Section 4 will describe the language modelling approach in details and present examples of the method in action. The description is followed by some of the smoothing methods, whose performance will be discussed at the end of the section.