{'fraction_non_alphanumeric': 0.01775147928994083, 'fraction_numerical': 0.0, 'mean_word_length': 5.0, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Outlier Detection in Text Data: An Unsupervised Method Based On Text Similarity and Density Peak


 Outlier detection is problematic in the text data processing. In this paper, a novel method for detecting outliers in text data is proposed. To recognize the outliers, we concentrate on the similarity between every two documents and the density of similar documents. Afterward, an algorithm, which is based on these two elements, is written. We conducted a series of experiments on real datasets with different volumes to evaluate the performance of our proposed method. Experimental results obviously show that our method outperforms than a few previous methods especially when the volume of a dataset is medium. Furthermore, they represent that, we have no performance in one of the powerful outlier detection algorithms due to small datasets.