{'fraction_non_alphanumeric': 0.02495274102079395, 'fraction_numerical': 0.00831758034026465, 'mean_word_length': 5.1678321678321675, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Proposal : News Analysis for Social Sciences

The Lydia system [2, 3, 14, 17, 19, 20, 21, 23] analyzes spatial, temporal, and linguistic statistics of named entity occurrences in news text. As a result, it provides the user with a bird’s-eye view of the media coverage of a specific entity. This system can be a valuable tool for social sciences such as political science, history, and economics. A time series of the evolution of associations of an entity with other entities and traits is helpful in testing hypotheses in political science and history. A map showing the spatial distribution of entity popularity, sentiment polarity, and subjectivity towards that entity in the country is useful in reasoning about the level of success of a political campaign. Sentiment time series can be incorporated into studies of media influence and used to study voter behavior in response to local or national media coverage. I have worked on projects related to news and text analysis starting from Spring 2006, and have two relevant papers: “Concordance-based entity-oriented search” [2] and “International sentiment analysis for news and blogs” [3]. My plan for my thesis work is to develop Lydia into a general-purpose visual text analytics engine useful for a broad range of applications in social sciences. Making Lydia viable as a source of high-quality scientific data to feed into social science models and providing a convenient and high-performance user interface requires substantial technical efforts. Instead of trying to re-engineer Lydia’s four-year-old code written by several generations of graduate students and work around its performance bottlenecks, we decided to build a new scalable text processing engine utilizing modern distributed computation technologies such as Hadoop [8] map-reduce [9] library. My role in this process encompasses the design and implementation of our new data aggregation and processing architecture, providing a framework for interactions between various components of the new system such as text and derived statistics processing phases, the on-demand data retrieval server, the new user interface web application, and experimental components aimed at improvement of individual analysis phases. I am also responsible for establishing and upholding code quality, design, implementation, and testing standards for our new system. The design of this distributed system is significantly more than a mere re-implementation of the previous Lydia system. Our new system is a data mining platform prepared to deal with a two order of magnitude larger text volume and offers an extensible statistics extraction