{'fraction_non_alphanumeric': 0.019793459552495698, 'fraction_numerical': 0.0068846815834767644, 'mean_word_length': 5.4972067039106145, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Latent Semantic Indexing with selective Query Expansion

This thesis outlines research and experiments in Information Retrieval, specifically with the search method Latent Semantic Indexing (LSI). The design and implementation of a framework for performing LSI queries on large text databases with a focus on machine learning is documented. Also described and analyzed is our participation in TREC in 2010 and 2011. Finally, we describe our next LSI-based task, Cyberbullying detection, which involves applying our system to Internet-chat-based datasets to discover instances of online harassment. The results of our research show that EDLSI is an effective technique for E-Discovery. Preliminary observations also show promise for EDLSI’s effectiveness in Cyberbullying detection. We have shown that selective query expansion can be a useful mechanism for improving retrieval results when a specific initial query is provided. We believe that queries that are stated in general terms, however, may suffer from “expansion in the wrong direction” when certain iterative approaches to incorporating relevance feedback information are added into the search process.