{'fraction_non_alphanumeric': 0.02043506921555702, 'fraction_numerical': 0.0, 'mean_word_length': 5.145748987854251, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Research Proposal : Automatic Grammar Induction and Semantic Discovery

I propose research in automatic grammar induction and discovery of semantic relationships. The former has applications in the study of language similarity, language acquisition, and numerous other fields within linguistics and language technology; the latter can improve information retrieval, information extraction, dialog systems, and automatic spelling and grammar checkers. Both are active research topics in the cross-section between linguistics and computer science, and I feel that the two can be combined into something that is better than the sum of its parts. To this end, I will construct and evaluate a system capable of extracting grammatical and semantic information from unannotated corpora (and thereby for any language). Work on the grammar portion of this system has already begun, the results so far are encouraging, and it appears that, to a certain extent, I have independently arrived at some of the same ideas as those behind the Adios system, which is developed jointly by the Cornell and Tel Aviv universities and which I will study further. In my work, I will consider not only phrase grammars, but also rules for phonology, morphology, and discourse. I will then extend this system to also extract semantic information from the corpora presented to it, allowing a deeper and more thorough analysis. Here, I will explore to what extent this can be done with little or no additional data outside the corpora themselves.