{'fraction_non_alphanumeric': 0.02478448275862069, 'fraction_numerical': 0.003232758620689655, 'mean_word_length': 5.319727891156463, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Document Retrieval by Similarity : An Application of Probabilistic Latent Semantic Analysis ( PLSA )

In this project, we explored using Probabilistic Latent Semantic Analysis (PLSA) technique to model a large collection of documents. Based on such a model, we also investigated the possibility of an interface that allows a user to 1) observe and explore the document collection on macroscopic level; 2) conduct specific search based on document similarity. PLSA’s performance on standard information retrieval (IR) tasks has been well documented [2]. The emphasis of this project is to build an application based on PLSA model to help end-users to explore and search in large collection of documents. Many practical issues arise during this attempt, such as computational efficiency and interpretation of learned model parameters. This report summarizes experience gained and lessons learned from building such an application.