{'fraction_non_alphanumeric': 0.023232984293193717, 'fraction_numerical': 0.001963350785340314, 'mean_word_length': 5.8389261744966445, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a Universal Analyzer of Natural Languages

Developing NLP tools for many languages involves unique challenges not typically encountered in English NLP work (e.g., limited annotations, unscalable architectures, code switching). Although each language is unique, different languages often exhibit similar characteristics (e.g., phonetic, morphological, lexical, syntactic) which can be exploited to synergistically train analyzers for multiple languages. In this thesis, we advocate for a novel language-universal approach to multilingual NLP in which one statistical model trained on multilingual, homogenuous annotations is used to process natural language input in multiple languages. To empirically show the merits of the proposed approach, we develop MALOPA , a language-universal dependency parser which outperforms monolingually-trained parsers in several low-resource and high-resource scenarios. MALOPA is a greedy transitionbased parser which uses multilingual word embeddings and other language-universal features as a homogeneous representation of the input across all languages. To address the syntactic differences between languages, MALOPA makes use of token-level language information as well as language-specific representations such as fine-grained part-of-speech tags. MALOPA uses a recurrent neural network architecture and multitask learning to jointly predict POS tags and labeled dependency parses. Focusing on homogeneous input representations, we propose novel methods for estimating multilingual word embeddings and for predicting word alignments. We develop two methods for estimating multilingual word embeddings from bilingual dictionaries and monolingual corpora. The first estimation method, multiCluster, learns embeddings of word clusters which may contain words from different languages, and learns distributional similarities by pooling the contexts of all words in the same cluster in multiple monolingual corpora. The second estimation method, multiCCA, learns a linear projection of monolingually trained embeddings in each language to one vector space, extending the work of Faruqui and Dyer (2014) to more than two languages. To show the scalability of our methods, we train multilingual embeddings in 59 languages. We also develop an extensible, easy-to-use web-based evaluation portal for evaluating arbitrary multilingual word embeddings on several intrinsic and extrinsic tasks. We develop the conditional random field autoencoder (CRF autoencoder) model for unsupervised learning of structured predictors, and use it to predict word alignments in parallel corpora. We use a feature-rich CRF model to predict the latent representation conditional on the observed input, then reconstruct the input conditional on the latent representation using a generative model which factorizes similarly to the CRF model. To reconstruct the observations, we experiment with a categorical distribution over word types (or word clusters), and a multivariate Gaussian distribution that generates pretrained word embeddings.