{'fraction_non_alphanumeric': 0.017101710171017102, 'fraction_numerical': 0.0, 'mean_word_length': 5.390804597701149, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semantic clustering approach for documents in distributed system framework with multi-node setup

Today's era is rather called big data era, data starts growing from different sources of web and such scalable data is very hard to manage with the existing frameworks and technologies. Wikipedia is a content management system where the article posted has a number of source documents. Perhaps, it is very difficult to search an exact relevant document for selected content in Wikipedia article as it has too many sources such as primary, secondary and tertiary. In order to search and retrieve relevant document in the growing content and references, clustering of documents using similarity analysis is very much essential. The existing system offers a clustering technique based on term and inverse term frequency (TfDf) scoring method. This work proposes a new clustering method for distributed framework called semantic agglomerative hierarchical (SHA) clustering algorithm. The performance testing, evaluation is implemented in multinode environment. The metrics such as recall and precision are calculated.