{'fraction_non_alphanumeric': 0.01929260450160772, 'fraction_numerical': 0.0006430868167202572, 'mean_word_length': 5.53781512605042, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Selecting Effective Expansion Terms for Better Information Retrieval

Automatic Query expansion is a well-known method to improve the performance of information retrieval systems. In this paper, we consider methods to extract the candidate terms for automatic query expansion, based on co-occurrence information from psuedo relevant documents. The objective of the paper is: to present to user different ways of selecting and ranking co-occurring terms and to suggest use of information theoretic measures for ranking the co-occurring terms selected, in order to improve retrieval efficiency. Specifically in our work, we have used two information theoretic measures: Kullback-Leibler divergence (KLD) and a variant of KLD. These measures are based on relative entropy between top documents and entire collection. We have compared the retrieval improvement achieved by expanding the query with terms obtained with different methods belonging to both approaches (co occurrence based and information theoretic). Experiments have been performed on TREC-1 data set. Intensive experiments have been done to select suitable parameters used in automatic query expansion, such as number of top n selected documents and the number of terms selected for expansion. Results suggest that firstly considerable improvements can be achieved if co-occurring terms are selected properly by considering different options available for selecting them. Secondly, information theoretic measures applied over co-occurring terms can be helpful in improving retrieval efficiency.