{'fraction_non_alphanumeric': 0.01683087027914614, 'fraction_numerical': 0.0, 'mean_word_length': 5.32987012987013, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sampling and text classification techniques for data mining

This dissertation addresses the issues on how to improve the performance of data mining from transaction and text databases. We propose a novel sampling technique to achieve a trade-off between speed and accuracy for mining transactional databases. In addition, we discuss how to use a thesaurus-based hierarchical classification algorithm to categorize text documents. 
The sampling algorithm that we propose is called FAST (Finding Associations from Sampled Transactions). FAST is designed to discover association rules in large databases that exceed specified levels of confidence and support while avoiding the expensive scans of the entire database that are required by classical association rule mining algorithms. FAST is distinguished from previous sampling-based algorithms by its novel two-phase approach to sample collections. In Phase I a large initial sample of transactions is collected and used to quickly and accurately estimate the support of each individual item in the database. In Phase II these estimated supports are used to either trim “outlier” transactions or select “representative” transactions from the initial sample, thereby forming a small final sample that more accurately reflects the statistical characteristics of the entire database. The expensive operation of discovering association rules is then performed on the final sample. Empirical study has shown that FAST can achieve a very good speed-accuracy trade-off. 
In the implementation of the HITEC (HIerarchical TExt Classification) algorithm, a concept hierarchy is constructed from an electronic thesaurus, where concepts are linked by parent-child relationships. The concept hierarchy is regarded as a primitive class hierarchy, with each concept corresponding to a class. One of the features of HITEC is that it is concept-oriented. In order to determine the topics more accurately, the classification procedure tries to identify the meanings of keywords, or concepts, in documents before they are mapped to the class hierarchy. Another distinctive feature of HITEC algorithm is that it generates the class hierarchy from a thesaurus, thus the semantic relationships among classes are fully preserved. We also propose a method to compute the weights in class representative vectors and show how they can be dynamically adjusted to reflect the topics of the documents in each class.