{'fraction_non_alphanumeric': 0.02315160567587752, 'fraction_numerical': 0.008961911874533234, 'mean_word_length': 5.504854368932039, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Efficient pre-processing for enhanced semantics based distributed document clustering

Document clustering, one of the traditional data mining techniques, is an unsupervised learning paradigm where clustering methods try to identify inherent groupings of the text documents, so that a set of clusters is produced in which clusters exhibit high intra-cluster similarity and low inter-cluster similarity. The importance of document clustering emerges from the massive volumes of textual documents created. For distributed document clustering, we experimented on Hadoop and MapReduce open source platforms. We tested on two datasets consisting of 20000 (20-NewsGroup) and 21578 (Reuters) documents. We focused on efficient pre-processing wherein we applied syntactic and semantics tagging to the dataset. Here, we have suggested syntactic and semantic parsing i.e. POS tagging, and word sense disambiguation for better understanding of synonymous words and hyponymy words using WordNet. Porter's Stemming algorithm is revised and compared with traditional algorithm. We also suggest different order of pre-processing steps. We performed experiments to generate various clusters with different count of iterations for refining the clusters. Also, for monitoring the scale-up of Hadoop we added nodes and compared the time taken for clustering.