{'fraction_non_alphanumeric': 0.03833652804400367, 'fraction_numerical': 0.01758479873322777, 'mean_word_length': 4.388414907947912, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bilingual Dictionary Induction for Bantu Languages

We present a method for learning bilingual translation dictionaries between English and Bantu languages. We show that exploiting the grammatical structure common to Bantu languages enables bilingual dictionary induction for languages where training data is unavailable.


Introduction
Bilingual dictionaries mostly exist for resourcerich language pairs, for example, English-German and English-Chinese (Koehn and Knight, 2002;Haghighi et al., 2008;Ammar et al., 2016;Faruqui and Dyer, 2014). Such dictionaries are useful for many natural language processing (NLP) tasks including statistical machine translation, cross-lingual information retrieval, and cross-lingual transfer of NLP models such as those for part-of-speech tagging and dependency parsing (Täckström et al., 2012;Guo et al., 2015;Gouws and Søgaard, 2015). In this paper, we consider the task of bilingual dictionary induction for English and Bantu languages. Bantu languages are a family of over 300 1 mutually intelligible languages spoken over much of central and southern Africa, see the map 2 in Figure  1 (Guthrie, 1948;Nurse and Philippson, 2003). About a third of Africans speak a Bantu language as their native language (Nurse and Tucker, 2001). The most widely spoken, with over 10 million speakers 3 each, include Swahili (Kenya, Uganda), Shona (Zimbabwe), Zulu (South Africa), and Kinyarwanda (Rwanda, Uganda). As with other low resource languages, labeled data for Bantu languages is scare. We seek to exploit the Bantu grammar structure to mitigate lack of labeled data. More specifically, we ask the following question: given a small bilingual dictionary between English and one Bantu language, L bantu1 , can we 1) infer missing entries in the English − L bantu1 dictionary 2) generate a new bilingual dictionary English − L bantu2 for another Bantu language for which labeled data is unavailable. To answer this question we propose an approach based on distributed representations of words (Turney and Pantel, 2010;Mikolov et al., 2013a). The first step is to create a vector space for each language, derived from a text corpus for the language. Notice that these text corpora need not be aligned. The second step is to perform dictionary induction by learning a linear projection, in the form of a matrix, between language vector spaces (Mikolov et al., 2013b;Lazaridou et al., 2015). Our key insight for Bantu languages is that one can create a single vector space for them, obviating the need for learning a projection matrix for each Bantu language. This means we only need to learn a single projection matrix, for inducing multiple English to Bantu bilingual dictionaries, using the small bilingual dictionary English−L bantu1 . Additionally, we modify the corpus corresponding to L bantu2 to have a greater vocabulary intersection with L bantu1 . This step is inspired by the extensive use of bases and affixes, common to Bantu languages. Words with the same meaning often differ only in the affixes with the base being similar or the same. We therefore use edit distance to replace some fraction of the words of L bantu2 with similar words in L bantu1 .
Contribution. 1). Unsupervised Bantu language dictionary induction: To the best of our knowledge, our work is the first effort to create bilingual dictionaries for Bantu languages using unsupervised machine learning methods. 2) Data: We collect corpora for seven Bantu languages. Having had access to a first language speaker of two Bantu languages, we obtained labeled which we make available along with the corpora, for further research into NLP for Bantu languages. 3) Dictionary induction almost from scratch: We propose a method for dictionary induction that only requires training data in one of the Bantu languages. Our experiments show the potential of our approach.

Distributed Representation
Distributed representations of words, in the form of real-valued vectors, encode word semantics based on collocation of words in text (Turney and Pantel, 2010;Mikolov et al., 2013a;Ammar et al., 2016). Such vector representations have been shown to improve performance of various NLP tasks including semantic role labeling, partof-speech tagging, and named entity recognition (Collobert et al., 2011). In this work we use the skip-gram model with negative sampling to generate word vectors (Mikolov et al., 2013a). It is one of the most competitive methods for generating word vector representations, as demonstrated by results on a various semantic tasks (Baroni et al., 2014;Mikolov et al., 2013b).

Bilingual Dictionary Induction
To induce a bilingual dictionary for a pair of languages, we use the projection matrix approach (Mikolov et al., 2013b;Lazaridou et al., 2015). It takes as input a small bilingual dictionary containing pairs of transla-tions from the source language to the target language. Training data is comprised of vector representations of word pairs , where x i ∈ R s is the vector for word i in the source language, and y i ∈ R t is the vector for its translation in the target language. At test time, we predict the target word translations for new source language words, D te = {x j } n i=1 where x j ∈ R s . In our case, the source language is a Bantu language and the target language is English.
This approach assumes that there is linear relationship between the two vector spaces. Thus, the learning problem is to find a matrix W that maps a source language word vector x i to the vector of its translation y i in the target language. As in , we use an l2-regularized least squares error to learn the projection matrix W. (1) where X and Y are matrices representing the source and target vectors in the training data, respectively. For a new Bantu word whose vector representation is x j ∈ R s , we map it to English by computingŷ j =Ŵx j whereŷ j ∈ R t , and then finding the English word whose vector representation is closest toŷ j , as measured by the cosine similarity distance metric.

Single Projection Matrix
We hypothesize that we only need to learn one projection matrix, W in Equation 1. Our labeled data is a small bilingual dictionary English − L bantu1 , between English and a Bantu language L bantu1 . We would like to be able to infer missing entries in the English − L bantu1 dictionary, and to generate a new dictionary, English − L bantu2 , a language pair for which labeled data is unavailable. The core idea is to create only one vector space for the two Bantu languages. First we generate a lexicon, lex b2 , containing words that appear in the corpus of L bantu2 . Next, for each w ∈ lex b2 we find all words in lex b1 , the lexicon of language L bantu1 , whose edit distance to w is a small value Φ 6 . Thus, each word w ∈ lex b2 has a list of words from lex b1 , S w = {w1 ∈ Lex b1 : editditance(w, w1) <= Φ}. We then go through corpus C bantu2 and with probability Π 7 , we replace word w ∈ lex b2 with one of its cross-lingually similar words in S w , random selection is used to pick the replacement word from S w . This process creates an updated corpuŝ C bantu2 . Concatenating the Bantu corpora gives: 6 In the experiments we use Φ = 1. 7 In the experiments we use Π = 0.5. people  ovanhu  aantu  return  aluka  galuka  child  okaana  omunona  things  oinima  iinima  ears  omakutwi  omakutsi  women  ovakainhu  aakiintu  tongue  elaka  elaka  darkness  omulaulu  omilema  feet  eemhadi  oompadhi  sins omatimba oondjo  C bantu = C bantu1 ∪Ĉ bantu2 . Applying the skipgram model to C bantu , generates word vectors, every w i ∈ C bantu has a vector x i ∈ R s . For English, we use the 300-dimensional pre-trained vectors trained the Google News dataset 8 , so that every English word has a vector y i ∈ R t . Finally, we W using the training data, D tr = {x i , y i } m i=1 . At test time, we predict the target word translations for unseen Bantu words, D te = {x j } n i=1 , which can either be in language L bantu1 or L bantu2 .

Experimental Evaluation
Data. We crawled Bible.com for bibles of 9 languages, 7 Bantu and 2 Indo-European, Italian and English. The latter were used for comparison. Corpora statistics are shown in Table 1. In our experiments, we focused on Kwanyama, spoken Namibia and Angola, as we had access to a first language speaker who could annotate data. The last column of Table 1 shows the vocabulary intersection between Kwanyama and other languages. The language with the most words in common with Kwanyama is Ndonga, spoken in Namibia, with an 11% vocabulary overlap. As expected, the Indo-European languages overlap the least with Kwanyama and the few words they have  in common have different meanings. For example: The word "male" in English refers to gender, in Kwanyama "male" means "tall" or "deep". Our Kwayama first language speaker also has five years of formal training in Ndonga, which is a dialect of the same language as Kwanyama. We therefore focus on these two Bantu languages in our experiments. Table 2 shows some examples of English, Kwanyama and Ndonga translations. Details of the training and test data are shown in Table 3. For all languages, we used 300-dimensional word vectors.
Results. Table 4 shows the main results in terms of precision at top-k, the last column, RD = 0.10 shows precision at the value of k which yields random chance of 10% precision. The top two rows show the results of bilingual dictionary induction between English and Kwanyama. We compare the projection matrix approach, EN-KW, to random chance, EN-KW (RD). We can see that EN-KW far outperforms chance. This result is promising given that our annotator only generated about 2, 142 labeled examples. In particular, English-Italian (EN-IT) with a larger dictionary of 5, 000 word pairs, produced by , achieves similar numbers, however it is worth noting that the EN-IT test data set is also much larger. For the English-Ndonga, EN-ND, language pair, we have no labeled data. We consider three cases: 1) EN-ND (J-KW), for this case, we concatenate the Kwanyama and Ndonga corpora and use the EN-KW training data to induce the EN-ND dictionary. 2) EN-ND (J-IT), we concatenate the Italian and Ndonga corpora and use the EN-IT training data to induce EN-ND dictionary. 3) EN-ND (J-KW-R), this is our approach where we first modify the Ndonga corpus to look more like Kwanyama , which is to be expected because ND, Ndonga, a Bantu language is much more similar to KW, Kwanyama than to the Indo-European language, IT, Italian. Figure 2 shows the top-k precision trends for various values of k. For the EN-KW pair, left of Figure 2, there is a bigger gap between EN-KW and random chance EN-KW (RD). On the other hand, for the EN-ND pair, the right of Figure  2, the gap between our approach EN-ND (J-KW-R) and random choice, EN-ND (RD) is smaller. However, it is also clear that the precision at top-k trend is much better when we make use of training data from Kwanyama EN-ND (J-KW-R), instead of training data from Italian EN-ND (J-IT). This result is encouraging for future work towards inducing accurate bilingual dictionaries for Bantu languages without labeled data. Future directions include collecting more training data from popular Bantu languages such as Swahili and Zulu; proposing alternative methods to dictionary induction; and inducing dictionaries for more Bantu languages.

Conclusion
In prior work, bilingual dictionary induction has been studied mostly for resource rich languages. (Lazaridou et al., 2015;Upadhyay et al., 2016;Faruqui and Dyer, 2014;Mikolov et al., 2013b;Ammar et al., 2016;Haghighi et al., 2008). We have introduced an approach where we create one vector space for Bantu languages in order to ex-ploit labeled data available for one language but not for another. Given that there are over 300 Bantu languages, and not all of them have training data, we believe approaches that rely on their shared grammar will be important for bringing NLP methods to this family of languages.