{'fraction_non_alphanumeric': 0.03288364249578415, 'fraction_numerical': 0.012264295569523225, 'mean_word_length': 4.785809312638581, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EnAsCorp1.0: English-Assamese Corpus

The corpus preparation is one of the important challenging task for the domain of machine translation especially in low resource language scenarios. Country like India where multiple languages exists, machine translation attempts to minimize the communication gap among people with different linguistic backgrounds. Although Google Translation covers automatic translation of various languages all over the world but it lags in some languages including Assamese. In this paper, we have developed EnAsCorp1.0, corpus of English-Assamese low resource pair where parallel and monolingual data are collected from various online sources. We have also implemented baseline systems with statistical machine translation and neural machine translation approaches for the same corpus.


Introduction
Assamese (also called as Axomiya) is the predominant language of the Indian state of Assam. It is one of the 23 official languages (22 Indian languages and additional language English) recognized by the Republic of India 1 . It is the most widely used language of the entire north-eastern region (also known as the seven sister states) of India. Assamese is spoken by about 14 million speakers (Barman et al., 2014), they are known as Assamese or Axomiya/Asomiya people. It belongs to the Indo-Aryan language family, and its script is evolved from the Gupta script (Dutta, 2019), also known as Assamese-Bengali script (Mahanta, 2012). In today's digital world, there is a demand for automatic translation of English to Assamese and vice-versa. In a multilingual country like India, machine translation (MT) plays a vi-1 https://www.mhrd.gov.in/sites/upload_ files/mhrd/files/upload_document/languagebr. pdf tal role to minimize the language barrier via automatic translation among human spoken languages (also known as natural languages). For all official modes of communication, Hindi and English languages are used by the government of India. Therefore, to facilitate greater access among the local community, it is necessary to translate the official documents like orders, notices, messages released by the government of India into the regional language of Assam. Moreover, English is a high resource and widely accepted language all over the globe. To establish a better communication in the native languages at the national or international level, MT system for the English-Assamese pair is very much essential. But due to the non availability of a suitable corpus, the MT system for this language pair still remains in beginning stage (Barman et al., 2014;Baruah et al., 2014).
In MT, based on the availability of corpus resource, there are two categories of natural languages: high and low resource languages. High resource languages are those languages which are resource-rich languages like English, German, French, and Hindi. On the other hand, low resource languages are resource-poor like many Indian languages especially found in the north-eastern region of India like Assamese, Boro, Khasi, Manipuri, Kokborok, and Mizo. Data scarcity is one of the major problems in MT for low resource language scenarios. For English-Assamese, there is a lack of standard parallel data and monolingual Assamese data. Because of this reason, English-Assamese can be categorized under low resource pair. The word order and script of both languages are completely different. Unlike English, Assamese follows the subject-objectverb (SOV) word order. Figure 1 depicts an example sentence, where an Assamese sentence is shown with its English translation and transliteration. Moreover, Assamese is known as mor-  Figure  2 presents an example of Assamese morpheme. There are total of 52 letters in the Assamese alphabet comprising of 41 consonants and 11 vowels. Assamese script and numerals are very different from English and identical to the Bengali language except for the two letters, ৰ (ro) and ৱ (vo) (Mahanta, 2012) In this paper, our focus is to create an English-Assamese corpus that we hope that it will be of benefit to the research community.

Role of Corpus in MT
MT has evolved from rule-based methods to a corpus-based approach, where any source language can be translated to any target language. Therefore, MT reached a language-independent stage. Under the corpus-based approach, neural machine translation (NMT) achieves a state-ofthe-art over the contemporary statistical machine translation (SMT). But both the approaches need sufficient training corpus which is a challenging issue, especially in the scenario of low resource languages. In MT, there are two types of datasets: parallel and monolingual corpora. The parallel corpus consists of aligned sentences, which are structured set of translated texts between source and target languages. It is required for both SMT and NMT based approaches (Koehn, 2010;Devlin et al., 2014). Such a parallel data is important for the model to learn word alignment between source and target sentences by estimating parameters. This is known as adequacy factor of a good translation. By increasing the amount of parallel corpus, the translation accuracy can be improved in both SMT and NMT. For English-Assamese, it is a challenging task to obtain a parallel corpus since there is lack of standard parallel data. Apart from the parallel corpus, there is a need for monolingual corpus as well, which is used to produce a fluent translation of the target sentence. The fluency is another factor for a good translation. Unlike parallel corpus, monolingual corpus for English and Assamese are available in various online sources, but since Assamese is a low resource language the digitized monolingual corpus is difficult to find. The standard English monolingual data are available from the shared task of MT 2 , but there is lack of standard Assamese monolingual data. This is an another challenge to prepare an Assamese monolingual corpora from available online sources. The monolingual corpus plays a vital role in improving the translation quality for both SMT (Koehn, 2010) and NMT in low resource language scenarios (Sennrich et al., 2016).

Corpus Details
There are very limited available options for English-Assamese parallel data. However, several possible sources have been explored to collect parallel and monolingual data as discussed in Section 2.3 and depicted in Figure 3. The details of the data sources used are described below: • SEBA Multilingual Question Paper: Board of secondary education, Assam (also known as SEBA) is the government recognized authority for conducting the school level examination in Assam. The multilingual SEBA examination question paper written in (English, Assamese, Hindi, and Bengali) contains exact parallel English-Assamese sentences as shown in Figure 4. From here, we have collected 1,000 parallel sentences using two steps. Firstly, we extracted the text from the portable document format (PDF) files of the question paper using Google's optical character recognition (OCR) and stored them into text files and then in the second step, we have manually collected the parallel sentences from the generated text files. •

Monolingual Corpus
As we have mentioned in Section 2.1 that standard monolingual data for English is available. Therefore, we have focused on the preparation of Assamese monolingual data only and the same

English Assamese
Explain briefly in what circumstances was the "Gandhi-Irwin", pact signed ? Discuss its terms.    been collected from different online web pages/Blogs and Holy books namely, Bible, Quran and Gita. Additionally, we can use the Assamese data from the parallel corpus to increase the Assamese monolingual data.

Corpus Statistics
The overall corpus statistics are shown in Table  1. The total parallel and monolingual sentences 251,024 and 50,086. Basically, there are three types of dataset, training, validation and test set. The training set is required for learning the parameters during the training of a model. During training process, the validation dataset is used to check the performance of a model to select the optimum model generated. The test data is the unseen data which is used to verify the model performance once the training process is completed. For the MT system, the parallel data need to split into train, validation, and test data. Before the split, we have removed duplicate sentences and the count of total parallel sentences got reduced to 210,215. The statistics for train, validation and test data are presented in Table 3. In the test test, we have included 100 out of domain parallel sentences from different blogs/web pages. These 100 parallel sentences include simple sentences which are commonly used in day to day life.

Data Extraction Techniques
For scraping the data from various online sources, we have utilized Scrapy 4 an open source framework for web crawling. The xpath for each element is coded with a certain degree of generalization in order to replicate for multiple web pages. This helped us in crawling a large number of web pages and extracting useful information. In order to extract text from images as well as from the PDF files we employ the use of Google OCR 5 tools. The process is summarized in Figure 3.
As given in the Figure 3, we begin by feeding the URLs of the web pages. The PDF documents are downloaded and the Assamese raw text in the HTML files are extracted directly. According to the format of the data here, the textual data is obtained. If the data is in image or PDF format, it is treated as an image and the Google OCR engine is used to extract the text. The use of Google OCR helped us to extract the data from a wide range of sources without facing any font related issues. The 4 https://scrapy.org/ 5 https://cloud.google.com/vision/ conversion of PDF to images resolves the issues of different fonts, embedded texts or size. The obtained mono-lingual data is kept as is, but the parallel data is aligned by separating them into separate source and target files. This process of alignment and verification, took substantial human effort. Table 4 presents example parallel sentences from each individual sources. We have removed noise i.e. unwanted symbols, too many special characters (......., ####, $$$$), URLs, English text from Assamese text, blank lines, etc from the text. We did not remove punctuation marks and not performed lower-case conversion for the English sentences because if we do this then the contextual meaning of the sentence will be changed as shown in Table 2. Assamese language has single case letters unlike English and hence no case conversion is considered here. Moreover, we split large sentences (more than 120 words) at the end of sentence punctuation mark (।,?).

Domain Coverage
Our corpus EnAsCorp1.0 covers variety of domains namely, Holy books (Bible, Quran, Gita), daily usage, literature and general domains.

Baseline Systems
For the baseline systems, we have trained two models: phrase-based SMT (Koehn et al., 2003) and recurrent neural network (RNN) based NMT with attention (Bahdanau et al., 2015) to estimate benchmark translation accuracy for English to Assamese and Assamese to English translation respectively. We have used the developed dataset EnAsCorp1.0 and English monolingual data about 3 million sentences from WMT16 6 .

SMT Setup
We have trained the phrase-based SMT using the Moses 7 (Koehn et al., 2007) toolkit. GIZA++ and IRSTLM (Federico et al., 2008) are used to generate phrase pairs and language model following default settings.

NMT Setup
We have trained the NMT using OpenNMTpy toolkit 8 (Klein et al., 2017). The encoderdecoded architecture is of a two-layer network consists of long short term memory (LSTM) cell with 500 nodes in each layer and attention mechanism (Bahdanau et al., 2015) are used. The Adam optimizer with default learning rate and drop-outs 0.3 are used. We have employed pre-trained word vectors using Glove 9 (Pennington et al., 2014) by utilizing the monolingual data to capture semantic word relationships.

Results
To evaluate the performance of the baseline systems, we have used bilingual evaluation understudy (BLEU) (Papineni et al., 2002) automatic evaluation metric. Table 5 presents the results of both the systems. We have considered the average BLEU score of uni-gram, bigram, and tri-gram since the BLEU score tends to diminish after crossing the tri-gram score.

Conclusion and Future Work
In this paper, we have presented EnAsCorp1.0 for various tasks of NLP, mainly MT. We have developed both parallel and monolingual corpus for low resource English-Assamese pair. The dataset will be available here: https: //github.com/cnlp-nits/EnAsCorp1.0. We have evaluated baseline systems using SMT and NMT. It is observed that the performance accuracy of NMT is better than SMT. In the future, we will increase the size of the corpus and develop MT system by utilizing advanced deep learning techniques for the same pair. The proposed dataset will be available for the benefit of the research community.