{'fraction_non_alphanumeric': 0.03178262084381112, 'fraction_numerical': 0.01941883207599888, 'mean_word_length': 4.908790755262072, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
OdiEnCorp 2.0: Odia-English Parallel Corpus for Machine Translation

The preparation of parallel corpora is a challenging task, particularly for languages that suffer from under-representation in the digital world. In a multi-lingual country like India, the need for such parallel corpora is stringent for several low-resource languages. In this work, we provide an extended English-Odia parallel corpus, OdiEnCorp 2.0, aiming particularly at Neural Machine Translation (NMT) systems which will help translate English↔Odia. OdiEnCorp 2.0 includes existing English-Odia corpora and we extended the collection by several other methods of data acquisition: parallel data scraping from many websites, including Odia Wikipedia, but also optical character recognition (OCR) to extract parallel data from scanned images. Our OCR-based data extraction approach for building a parallel corpus is suitable for other low resource languages that lack in online content. The resulting OdiEnCorp 2.0 contains 98,302 sentences and 1.69 million English and 1.47 million Odia tokens. To the best of our knowledge, OdiEnCorp 2.0 is the largest Odia-English parallel corpus covering different domains and available freely for non-commercial and research purposes.


Introduction
Odia (also called Oriya) is an Indian language belonging to the Indo-Aryan branch of the Indo-European language family. It is the predominant language of the Indian state of Odisha. Odia is one of the 22 official languages and 14 regional languages of India. Odia is the sixth Indian language to be designated a Classical Language in India based on having a long literary history and not having borrowed extensively from other languages. 1 Odia is written in Odia script, which is a Brahmic script. Odia has its origins pinned to the 10th century. In the 16th and 17th centuries, as in the case of other Indian languages, Odia too suffered changes due to the influence of Sanskrit. 2 Odia is nowadays spoken by 50 million speakers. 3 It is heavily influenced by the Dravidian languages as well as Arabic, Persian, English. Odias inflectional morphology is rich with a three-tier tense system. The prototypical word order is subject-object-verb (SOV). In today's digital world, there has been a demand for machine translation systems for English↔Odia translation for a long time which couldn't have been fulfilled due to the lack of Odia resources, particularly a parallel corpus. Parallel corpora are of great importance in language studies, teaching and many natural language processing applications such as machine translation, cross-language information retrieval, word sense disambiguation, bilingual terminology extraction as well as induction of tools across languages. The Odia language is not available in many machine translation systems. Several researchers explored these goals, developing Odia resources and prototype machine translation systems but these are not available online and benefitting users (Das et al., 2018;Balabantaray and Sahoo, 2013;Rautaray et al., 2019). We have analysed the available English-Odia parallel corpora (OdiEnCorp 1.0, PMIndia) and their performance (BLEU score) for machine translation (Parida et al., 2020;Haddow and Kirefu, 2020). OdiEnCorp 1.0 contains Odia-English parallel and monolingual data. The statistics of OdiEnCorp 1.0 are shown in Table 1. In OdiEnCorp 1.0, the parallel sentences are mostly derived from the English-Odia parallel Bible and the size of the parallel corpus (29K) is not sufficient for neural machine translation (NMT) as documented by the baseline results (Parida et al., 2020) as well as attempts at improving them using NMT techniques such as transfer learning (Kocmi and Bojar, 2019). The recently released PMIndia corpus (Haddow and Kirefu, 2020) contains 38K English-Odia parallel sentences but it is mostly collected from the prime minister of India's official portal 4 containing text about government policies in 13 official languages of India. These points motivate us for building OdiEnCorp 2.0 with more data, covering various domains suitable for various tasks of language processing, but particularly for the building of an English↔Odia machine translation system which will be useful for the research community as well as general users for non-commercial purposes.

Data Sources
As there is a very limited number of online Odia resources available, we have explored several possible ways to collect Odia-English parallel data. Although these methods need a considerable amount of manual processing, we opted for them, to achieve the largest possible data size. In sum, we used these sources: • Data extracted using OCR, • Data extracted from Odia Wikipedia, • Data extracted from other online resources, • Data reused from existing corpora.
The overall process of the OdiEnCorp 2.0 is shown in Figure 1.

OCR-Based Text Extraction
Many books are translated in more than one language and they could serve as a reliable source to obtain parallel sentences, but they are unfortunately not digitized (Bakliwal et al., 2016;Premjith et al., 2016). OCR technology has improved substantially, which has allowed for large-scale digitization of textual resources such as books, old newspapers, ancient hand-written documents (Dhondt et al., 2017). That said, it should be kept in mind that there are often mistakes in the scanned texts as OCR system occasionally misrecognizes letters or falsely identifies text regions, leading to misspellings and linguistics errors in the output text (Afli et al., 2016). Odia language has a rich literary heritage and many books are available in printed form. We have explored books having either English and Odia parallel text together or books having both versions (English and Odia). We have used the study, translation, grammar, literature, and motivational books for this purpose, obtaining the source images either from the web, or directly scanning them ourselves.
We start with the image containing the Odia language text represented in the RGB color space. For the Odia text recognition, we use the "Tesseract OCR engine" (Smith, 2007) with several improvements in the pre-processing phase. First, we move from the traditional method which converts RGB to grayscale by taking the simple average of the three channels. We convert the RGB image into a grayscale image where R is the amount of red, G green and B blue color in a pixel.
To change the image further to black and white only, "Tesseract" uses the traditional binarization algorithm called "Otsu". We use instead the "Niblack and Sauvola threshold algorithm" which we found to give better results. The advantage of the Niblack algorithm is that it slides a rectangular window through the image (Smith, 2007). The center pixel threshold T is derived from the mean m and variance s values inside the window.
where k is a constant set to 0.8. "Niblack" can create noise in some areas of the image, so we further improve it by including the "Sauvola" algorithm. Thus the modified formula is: where R is the dynamics of standard deviation, a constant set to 128. This formula will not detect all the images of documents. So the normalized formula we have implemented is: However, some black pixels vanish during these processes which may lead to erroneous character recognition, so we use Dilation (Gaikwad and Mahender, 2016) to join areas which got accidentally disconnected, see Figure 3 for an illustration.
Because dilation sometimes produces too many black pixels, we further apply "Erosion" (Alginahi, 2010) as illustrated in Figure 4. Finally, Figure 2 illustrates a sample of the scanned image containing parallel Odia-English data and the extracted text.

Odia Wikipedia
The Odia Wikipedia started in 2002 and serves as a good source for Odia-English parallel data. The following steps were performed to obtain parallel sentences from Odia Wikipedia, with more details provided in the sections below:

Additional Online Resources
Finding potential parallel texts in a collection of web documents is a challenging task, see e.g. (Antonova and Misyurev, 2011;Kúdela et al., 2017;Schwenk, 2018;Artetxe and Schwenk, 2019). We have explored websites and prepared a list of such websites which are potential for us to collect Odia-English parallel data. The websites were then crawled with a simple Python script. We found Odisha's government portals of each district (e.g. Nayagarh district 5 ) of Odisha containing general information about the district in both English and Odia version.
Analyzing extracted text, we found a few cases where English was repeated in both sides of the website. We have aligned the extracted text manually to obtain the parallel text.
We also extracted parallel data from the Odia digital library "Odia Virtual Academy", 6 an Odisha government-initiated portal to store treasures of Odia language and literature for seamless access to Odia people staying across the globe.
The web page provides tri-lingual books (tribal dictionary 7 containing common words and their translations in English and Odia) and we extracted the English-Odia sentence pairs from it.

Reusing Available Corpora
Finally, we included parallel data from OdiEnCorp 1.0 and PMIndia (Parida et al., 2020;Haddow and Kirefu, 2020). Both corpora contain pre-processed English-Odia parallel sentences. The statistics of these corpora are available in Table 2.

Data Processing
The data collected from different sources were processed to achieve a unified format.

Extraction of Plain Text
When utilizing online resources, we used a Python script to scrape plain text from HTML pages.

Manual Processing
After analyzing the raw data extracted using the OCR-based approach, we found a few errors such as unnecessary characters and symbols, missing words, etc. One of the reasons of poor OCR performance was the fact that some images were taken using mobile phones. In later processing, we always used a proper scanner. We decided for manual correction of such entries by volunteers whose mother tongue is Odia. Although this task is time-consuming and tedious, the result should be of considerably better quality and much more suitable for machine translation and other NLP tasks. Four volunteers worked part-time (2-3 hours daily) for four months on scanning of books, extracting data from the scanned images using OCR techniques, collecting data from online as well as offline sources, and post-editing all the data collected from different sources.

Sentence Segmentation
All sources that come in paragraphs (e.g. Wikipedia articles or books) had to be segmented into sentences. We considered full stop (.) as of the end of the sentence for English and Odia Danda or Purnaviram (|) as of the end of the sentence for Odia language.

Sentence Alignment
For some sources, the alignment between English and Odia sentences was straightforward. Sources like Odia Wikipedia posed a bigger challenge, because the texts in the two languages are often created or edited independently of each other. To achieve the best possible parallel corpus, we relied on manual sentence alignment. In this process, we had to truncate or remove several few sentences in either of the languages in order to reach exactly 1-1 aligned English-Odia sentence pairs.

Domain Coverage
The

Final Data Sizes
The composition of OdiEnCorp 2.0 with statistics for individual sources is provided in

Baseline Neural Machine Translation
For future reference, we provide a very baseline experiment with neural machine translation using OdiEnCorp 2.0 data.

Dataset Description
For the purpose of NMT training, we removed duplicated sentence pairs and shuffled the segments The training, dev and test set sizes after this processing are shown in Table 3.

Neural Machine Translation Setup
We used the Transformer model (Vaswani et al., 2018) as implemented in OpenNMT-py (Klein et al., 2017). 8 Subword units were constructed using the word pieces algorithm (Johnson et al., 2017). Tokenization is handled automatically as part of the pre-processing pipeline of word pieces. We generated the vocabulary of 32k sub-word types jointly for both the source and target languages, sharing it between the encoder and decoder. To train the model, we used a single GPU and followed the standard "Noam" learning rate decay, 9 see (Vaswani et al., 2017) or (Popel and Bojar, 2018) for more details. Our starting learning rate was 0.2 and we used 8000 warm-up steps. The learning curves are shown in Figure 5 and Figure 6

Results
We use sacreBLEU 10,11 for estimating translation quality. Based on the Dev 2.0 best score, we select the model at iteration 40k for EN→OD and at 30k for OD→EN to obtain the final test set scores.

Conclusion and Future Work
We presented OdiEnCorp 2.0, an updated version of Odia-English parallel corpus aimed for linguistic research and applications in natural language processing, primarily machine translation. The corpus will be used for low resource machine translation shared tasks. The first such task is WAT 2020 13 Indic shared task on Odia↔English machine translation. Our plans for future include: • Extending OdiEnCorp 2.0 with more parallel data, again by finding various new sources.
• Building an English↔Odia translation system utilizing the developed OdiEnCorp 2.0 corpus and other techniques (back translation, domain adaptation) and releasing it to users for non-commercial purposes.  • Promoting the corpus in other reputed machine translation campaigns focusing on low resource languages.

Acknowledgements
The work was supported by an innovation project (under an Inno-Suisse grant) oriented to improve the automatic speech recognition and natural language understanding technologies for German (Title: SM2: Extracting Semantic Meaning from Spoken Material funding application no. 29814.1 IP-ICT). In part, it was also supported by the EU H2020 project "Real-time network, text, and speaker analytics for combating organized crime" (ROXANNE, grant agreement: 833635) and by the grant 18-24210S of the Czech Science Foundation. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (projects LM2015071 and OP VVV VI CZ.02.1.01/0.0/0.0/16 013/0001781).