{'fraction_non_alphanumeric': 0.013733905579399141, 'fraction_numerical': 0.0, 'mean_word_length': 5.041450777202073, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Natural Language Processing for Information Retrieval

Human Language Technology has played a big role in implementing Latin based information retrieval systems. Two of the most sited techniques are stemming and truncation. Numerous studies have showed that the inflectional structure of words has a big impact on the retrieval accuracy of Latin-based languages information retrieval systems (IRS). Stemming or truncation is done for two principal reasons: the reduction in index storage required and the increase in performance due to the use of word variants. Several stemming algorithms were proposed for stemming text such as Porter for English. While these studies were concerned with Latin-based languages, only few studies give attention to the Arabic language. This paper we present a study of the Arabic language characteristics that can be useful to integrate in an information retrieval system and the kind of stemming techniques that can be used for the Arabic language. We used the .ae domain as a case study. We present some characteristics of this domain in our statistical experiments and the criteria we are using to evaluate Arabic search engines.