{'fraction_non_alphanumeric': 0.01633045148895293, 'fraction_numerical': 0.0009606147934678194, 'mean_word_length': 5.676282051282051, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploiting user-generated data for knowledge discovery and recommendation

The Internet and Web 2.0 have achieved a rapid growth and became ubiquitous in recent years. The advance in information technologies has also enabled users to generate data, implicitly or explicitly, on an unprecedented scale. Consequently, the need to discover and exploit new and useful knowledge from such data has also increased considerably. In this thesis, in this regard, we investigate user-generated data to discover interesting knowledge and enable better recommendation services. First, we tackle the problem of the location type classification using individual Twitter messages. We extend probabilistic text classification models to incorporate temporal features and user history information as probabilistic priors, and show that the proposed models can boost the classification accuracy effectively. Second, we study the problem of quantifying the notion of political legitimacy using collective Twitter messages for specific populaces. We design a framework that aggregates a large number of tweets into the final legitimacy score of a populace by leveraging probabilistic topic modeling and sentiment analysis technique. Our empirical evaluation on eight sample countries using related public tweets demonstrates that our proposed framework shows a strong correlation to results reported in political science literature. We also apply this framework to a traditional news media data set, and compare the results with Twitter data. Several interesting differences are discovered between these two medias for this quantification task of political legitimacy. Third, we study the problem of mining implicit user feedback in recommendation systems. In particular, we tackle the cold-start problem of video recommendation using users’ co-view information. We propose a classification framework to incorporate co-view information based on previously seen video pairs, and learn the weights of video attributes for ranking candidate videos to recommend, yielding encouraging recommendation results.