{'fraction_non_alphanumeric': 0.016666666666666666, 'fraction_numerical': 0.0007246376811594203, 'mean_word_length': 5.423255813953489, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Vector versus Tree Model Representation in Document Clustering 1

The most used method for text documents representation is based on words frequency vectors named VSM (Vector Space Model). This representation, based only on words, loses any “word context” information which is found in the document and therefore reduces the learning efficiency in the process of classifying/clustering. In this article we intend to make a comparison between the classical method of document representation and a method called STDM (Suffix Tree Document Model) based on representing documents in the Suffix Tree format. For the second method we proposed a new approach for documents representation by performing suffix tree only for any two documents. This approach is faster and it has lower memory consumption in comparison with the common approaches that represent all documents from data set in the same suffix tree. Also for this method we have proposed a formula for computing the similarity between documents, which improves substantially the clustering quality. This representation method was validated using two clustering algorithms, HAC Hierarchical Agglomerative Clustering and k -Medoids. In this context we experiment also the stemming influence in the document pre-processing step and highlight the difference between similarity or dissimilarity measures to find “closer” documents.