{'fraction_non_alphanumeric': 0.0202757502027575, 'fraction_numerical': 0.0, 'mean_word_length': 5.407792207792208, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Building Maximum Entropy Text Classifier Using Semi-supervised Learning

Over the recent years, text classification has become one of the key techniques for organizing information. Since hand-coding text classifiers is impractical and hand-labeling text is time and labor consuming, it is preferable to learn classifiers from a small amount of labeled examples and a large example of unlabeled data. In many cases, such as online information retrieval or database applications, such unlabeled data are easily and abundantly available. Although a lot of this kind of learning algorithms have been designed, most of them rely on certain assumptions, which are dependent on specific datasets. Consequently, the lack of generality makes these algorithms unstable across different datasets. Therefore, we favor an algorithm with as little dependence on such assumptions or as weak assumption as possible. The maximum entropy models (MaxEnt) offers a generic framework meeting this requirement. Built upon a set of features which is equivalent to undirected graphical models, it provides a natural leverage of feature selection. Most importantly, the only assumption made by MaxEnt is that the average feature values on labeled data give a reasonable (not too deviated) estimation of those values on both labeled and unlabeled data. This is a very weak statistical assumption, underpinning the generality of MaxEnt. Even if the assumption is not strictly satisfied in some situations, theoretical bounds are derived on the generalization error. Similar to soft-max regression, MaxEnt also employs a straightforward mechanism for multi-class classification. With its standard form equivalent to maximum likelihood estimation, there are numerous smoothing approaches proposed to overcome overfitting. Several algorithms for solving this convex optimization problem are also proposed, with different performance and constraints. The main focus of the project is how to incorporate unlabeled data into MaxEnt. As the standard MaxEnt does not perform satisfactorily, side information is considered to be added to MaxEnt, in forms of minimal spanning tree, k-nearest neighbor, multi-representation of examples, etc. Our initial experimental result suggests that MaxEnt with side information is a promising tool. Other promising research areas under this framework are also pointed out in this report, after a comprehensive survey on both learning with unlabeled data and MaxEnt.