{'fraction_non_alphanumeric': 0.03293596209689745, 'fraction_numerical': 0.019639309185388965, 'mean_word_length': 4.87629097440503, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Simple or Complex? Learning to Predict Readability of Bengali Texts

Determining the readability of a text is the first step to its simplification. In this paper, we present a readability analysis tool capable of analyzing text written in the Bengali language to provide in-depth information on its readability and complexity. Despite being the 7th most spoken language in the world with 230 million native speakers, Bengali suffers from a lack of fundamental resources for natural language processing. Readability related research of the Bengali language so far can be considered to be narrow and sometimes faulty due to the lack of resources. Therefore, we correctly adopt document-level readability formulas traditionally used for U.S. based education system to the Bengali language with a proper age-to-age comparison. Due to the unavailability of large-scale human-annotated corpora, we further divide the document-level task into sentence-level and experiment with neural architectures, which will serve as a baseline for the future works of Bengali readability prediction. During the process, we present several human-annotated corpora and dictionaries such as a document-level dataset comprising 618 documents with 12 different grade levels, a large-scale sentence-level dataset comprising more than 96K sentences with simple and complex labels, a consonant conjunct count algorithm and a corpus of 341 words to validate the effectiveness of the algorithm, a list of 3,396 easy words, and an updated pronunciation dictionary with more than 67K words. These resources can be useful for several other tasks of this low-resource language. We make our Code&Dataset publicly available at https://github.com/tafseer-nayeem/BengaliReadability} for reproduciblity.


Introduction
The term "Readability" measures how much energy the reader will have to expend in order to understand a writing at optimal speed and find interesting. Readability measuring formulas, such as Automated Readability Index (ARI) (Senter and Smith 1967), Flesch Reading Ease (Flesch 1948), and Dale-Chall formula (Dale and Chall 1948;Chall and Dale 1995) calculate a score that estimates the grade level or years of education of a reader based on the U.S. education system, which is illustrated in Figure 1. These formulas are still used in many widely known commercial readability measuring tools such as Grammarly 2 and Readable. 3 This measurement plays a significant role in many places, such as education, health care, and government (Grigonyte et al. 2014;Rets et al. 2020;Meng et al. 2020). Government organizations use it to ensure that the official texts meet a minimum readability requirement. For instance, the Department of Insurance at Texas has a requirement that all insurance policy documents have a Flesch Reading Ease (Flesch 1948) score of 40 or higher, which translates to the reading level of a first-year undergraduate student based on the U.S. education system. 4 A legal document which is hard to read can lead someone to sign a contract without understanding what they are agreeing to. Another common usage area is the healthcare sector to ensure the proper readability of the care and treatment documents (Grigonyte et al. 2014). Better readability will attract visitors or readers of different websites or blogs, whereas poor readability may decrease the number of readers (Meng et al. 2020). Readability measures are also often used to assess the financial documents such as annual reports of a company's economic performance so that the information is more transparent to the reader (Loughran and McDonald 2014). Dyslexia is a disorder that causes difficulties with skills associated with learning, namely reading and writing, which affects up to 20% of the general population. Readability formulas have been applied to measure the difficulty of reading texts for people with dyslexia (Fourney et al. 2018).
The scores from readability formulas have been generally found to correlate highly with the actual readability of a text written in the English language. The adaptation of readability formulas to no-English texts is not straightforward. Measuring readability is also essential for every non-English language, but not all of the readability formulas mentioned above are language-independent. These formulas require some resources like a 3000-word list, which is easily understandable by fourth-grade American students, syllable counting dictionary, stemmer, lemmatizer etc. Resource availability for Natural Language Processing (NLP) research is an obstacle for some low-resource-languages (e.g., Bengali). In this paper, we aim to develop a readability analysis tool for the Bengali Language. Bengali is the native language of Bangladesh, also used in India (e.g., West Bengal, Tripura) and has approximately 230 million native speakers. 5 Despite being the 7 th most spoken language in the world, Bengali suffers from a lack of fundamental resources for NLP. For a low resource language like Bengali, the research in this area so far can be considered to be narrow and sometimes incorrect. Islam, Mehler, and Rahman (2012); Sinha et al. (2012) tried to adapt the formula-based approaches used for the English language. Unfortunately, it isn't straightforward as these formulas are developed for U.S. based education system 6 and which predicts U.S. grade level of the reader. Since the Bangladeshi education system and grade levels 7 are different from U.S., therefore, the mapping is faulty and led to incorrect results. There is a strong relationship between reading skills and human cognition, which varies depending on different age groups (Riddle 2007). Therefore, to eliminate this incompatibility, in this paper, we map grade level to different age groups to present age-to-age comparison. Moreover, (Sinha and Basu 2016;Islam, Rahman, and Mehler 2014;Sinha et al. 2012) used traditional machine learning models to address this task on a very small scale dataset, which isn't publicly available. There are readability analysis tools available for English (Napolitano, Sheehan, and Mundkowsky 2015), Arabic (Al-Twairesh et al. 2016), Italian (Okinina, Frey, and Weiss 2020), and Japanese (Sato, Matsuyoshi, and Kondoh 2008) language. Unfortunately, no such tool is available for Bengali language that can validate the readability of a text. On the other hand, there is no largescale human annotated readability analysis dataset available to train supervised neural models for this extremely lowresource language. Our main contributions are summarized as follows: • To the best of our knowledge, we first design a comprehensive system for Bengali readability analysis, which includes datasets, human-annotated corpora and dictionar-ies, an algorithm, models, and a tool capable of proving in-depth readability information of the texts written in the Bengali language (see Figure 4).
• We correctly adopt document-level readability formulas traditionally used for U.S. based education system to the Bengali education system with a proper age-to-age comparison. We present a document level dataset consisting of 618 documents with 12 different grade levels to evaluate the adaptation of these formulas to the Bengali language (see Table 1).
• Due to the long-range dependencies of RNNs and the unavailability of large-scale human-annotated corpora, we further divide the document-level task into sentence-level and present a large-scale dataset consisting of 96,335 sentences with simple and complex labels to experiment with supervised neural models (see Table 2). We design neural architectures and make use of all available pre-trained language models of Bengali Language.
• We also propose an efficient algorithm for counting consonant conjuncts form a given word. We present a humanannotated corpus comprising 341 words with varying difficulties to evaluate the effectiveness of this algorithm.
• We design a readability analysis tool capable of analyzing text written in the Bengali language to provide indepth information on its readability and complexity which would be useful for educators, content writers or editors, researchers, and readers (see Figure 4).

Related works English and Other Languages
There has been a lot of work on readability analysis for the English language, some of which are: Automated Readability Index (Senter and Smith 1967), Flesch reading ease (Flesch 1948), Flesch-Kincaid grade level (Kincaid et al. 1975), Gunning Fog index (Kincaid et al. 1975), Dale-Chall formula (Dale and Chall 1948;Chall and Dale 1995), and SMOG grade (Mc Laughlin 1969). To calculate the readability score from these formulas, we need to extract various types of information from the input text, e.g., average sentence length, average word length, number of characters, number of syllables, and number of difficult words. The primary takeaway from the common formulas is simple. Using shorter sentences with shorter and more common words will typically make a piece of writing easier to understand. Other than using these traditional formulas, there have been a lot of recent work on readability analysis of English language (Vajjala and Lučić 2018; Dueppen et al. 2019;Rets et al. 2020;Meng et al. 2020), but especially in recent years, the scope of readability analysis research has been broadened towards other languages such as Russian

Bengali Language
Formula Based Approaches The research in this area for Bengali Language is narrow and sometimes faulty. Das and Roychoudhury (2006) created a miniature readability model with one and two parametric fits using multiple regression for Bengali text readability analysis. They used only seven paragraphs from seven documents. They used two parameters, such as average sentence length and number of syllables per 100 words, which is responsible for representing text as easy or difficult. Sinha et al. (2012); Sinha and Basu (2016) proposed two readability formulas for each Bengali and Hindi text using regression analysis, which are similar to readability formulas used for the English language. Readability formulas were also applied to Bangladeshi textbooks by Islam, Mehler, and Rahman (2012). They extracted three types of features from data such as lexical features, entropy-based features, and Kullback-Leibler Divergence-based features. Unfortunately, the scores returned from these readability formulas approximate what grade level of U.S. based education system someone will need in order to be able to read a piece of text easily. Since the Bangladeshi education system and grade levels are entirely different from the U.S., the mapping is faulty and led to incorrect results. In contrast, we map grade level to different age groups to present the age-to-age comparison for the readability formulas to eliminate the incompatibility.

Traditional Machine Learning Based Approaches
Sinha and Basu (2016) also used machine learning methods for Bengali readability classification, which are Support Vector Machine (SVM) and Support Vector Regression (SVR). They showed that features like average word length, number of consonant conjuncts play a significant role in Bengali readability analysis. Islam, Rahman, and Mehler (2014) used a combination of 18 lexical features and information-theoretic features to achieve a better score. Phani, Lahiri, and Biswas (2014) introduced the importance of inter-rater agreement study in the field of readability analysis of Bengali text. For agreement study, they used Cohen's kappa and Spearman's rank correlation coefficient. Recently, Phani, Lahiri, and Biswas (2019) proposed 11 readability measuring models based on regression analysis using 30 Bengali passages. They used features such as the presence of stop words, word sense, and POS tags. These prior works highlighted the importance of consonant conjunct for measuring readability. But they did not present any specific algorithm to compute consonant conjunct. Instead, in this paper, we present an efficient algorithm and human-annotated corpus to evaluate the effectiveness of the proposed algorithm. Another limitation of the works mentioned above is that their dataset is small scale and not publicly available. On the other hand, we present a large-scale dataset and design supervised neural models for Bengali readability prediction.

Dataset
We collect documents from several published textbooks and magazines from Bangladesh and India. These are the most common and very well-known among children and adults.
These documents usually were published after rigorous review and editorial process and widely read by various age groups. In this paper, we present two datasets for readability prediction. (1) Document-level dataset to experiment with formula-based approaches, (2) Sentence level dataset to train supervised neural models. Additional Sources We also collect documents (literature and articles) for both children and adults from various well known and popular sources, some of which are: Ichchhamoti 9 , Society for Natural language Technology Research 10 , Ebela 11 , Sananda 12 , and Bangla library. 13

Sentence-level Dataset
As we can see from Table 1, the document-level dataset is largely insufficient for training supervised neural models. Therefore, we further divide the documents into sentences to create a large-scale dataset for training neural models.
• Simple Documents: From the NCTB dataset, we select class 1 to 5 as simple documents as these documents are generally for 6 to 10 years old students. 7 Also, we take all the children type documents from the additional sources.
• Complex Documents: All adults type documents from additional sources, and we do not take any complex documents from the NCTB dataset.
Sentences from our simple documents are labeled as 'simple', and sentences from complex documents are labeled as 'complex'. So initially, we start with 40,917 simple sentences and 60,875 complex sentences. After carefully investigating the initial dataset, we found that the complex documents mostly contain complex sentences. However, some simple sentences also exist in complex documents and vice versa. To remove simple sentences from the complex set, we apply cosine similarity to every complex set sentences to every simple set sentences. We extract the sentences from the complex set, which has a semantic similarity score of 0.90 or more to the simple sentences. We manually recheck and annotate these extracted sentences to either simple or complex. Before measuring cosine similarity, we convert sentences to 300-dimensional vectors using a fastText pre-trained model for the Bengali language (Grave et al. 2018). It's important to note that these sentences we extracted are editor-verified and further annotated by us. Finally, after removing duplicate sentences, some simple sentences from complex sentences, and vise versa, we have 40,102 simple sentences and 56,233 complex sentences for training. While annotating, we also corrected the spelling mistakes to make the data clean. Table  2 indicates the summary of our sentence level dataset with train, dev, and test splits.

Experiments
We use our document level dataset to experiment with formula-based approaches and use the sentence-level dataset to train supervised neural models.

Formula-based Approaches
We select 10 documents (class 1 to 8, class 9/10, and class 11/12 in Table 3) from NCTB, and 4 documents (children 1 to adults 2 in Table 3) from additional sources. Due to the unavailability of the spoken syllable counting system for the Bengali language, we take a subset of the documents covering each class from the document level dataset. Flesch reading ease, Flesch-Kincaid grade level, Gunning Fog index, and SMOG grade formula require a common feature, which is the number of syllables. Counting syllables manually of all words from vocabulary is time-consuming. Google has provided NLP resources for various languages (e.g., Hindi, Urdu, Nepali, Sinhala, and Bengali). We use a pronunciation dictionary 14 for the Bengali language with 65,038 words. Although we use this dictionary, we have to manually count syllables for more than two thousand words, which are not present in that dictionary. We use the updated dictionary to experiment with formula-based approaches. Table 3 indicates the performance of formula-based approaches on our dataset. Here, we present a column BN age, which indicates the reader's age of the input documents. In Bangladesh, usually, children are admitted to Class 1 at the age of six, and 14 https://git.io/JJhdm complete their higher secondary education (Class 12) at the age of seventeen. 7 Therefore, we fill up the age of the first 10 documents in Table 3 according to this range. For Children 1 and Children 2, we set the age range 6-10 as children's literatures are created for 6 to 10 years old readers 15 , and 18 or more for Adults 1 and Adults 2 in Table 3. Instead of matching the U.S. and Bangladeshi grade level, we match 'U.S. age' and 'BN age' for measuring the performance of all the formulas.
Automated Readability Index (ARI) formula provides a score equivalent to a grade level which was developed to assess the readability of written materials used in the U.S. Air Force (Senter and Smith 1967). The formula uses long words and long sentences to calculate a readability score. From Table 3, age range 6-7 indicates first or second grade. 16 From the second column of Table 3, the first row where the input document is taken from Class 1, so we set the value of 'BN age' = 6. After applying ARI to the input document, we find ARI score = 1 ('ARI' column). Since ARI score of 1 indicates Kindergarten grade level, therefore, we set 'U.S. age' equals to "5-6". As we can see, this document is correctly classified because 6 is present in the range of 5-6.
Flesch reading Ease (FE) formula approximates a number indicating the difficulty of the input document. The higher the number, the easier it is to read the document (Flesch 1948). For example, if the score is between 90 to 100 for an input text, then this text is very easy to read and understandable by an average 11 years old reader. The formula focuses on the number of words, sentences, and syllables. In our case, we get the maximum value 40.92 for an input document from Bangladeshi Class 1 (6 years old). According to this formula, a score of 40 means the text is difficult to read and is understandable by U.S. college students (18 to 24 years old). The lowest value of this formula is 0, but we get negative values for 3 documents out of 14 documents. This formula is highly suited for the documents written in English language and used by the professional readability analysis tools like Grammarly 2 and Readable 3 . However, the formula is not suitable for the Bengali language because of the various linguistic difficulty present in the syllable counting system, and this is one of the main reasons we experiment with wide varieties of formulas.

Flesch-Kincaid (FK), Gunning Fog (GF), and SMOG
For all 3 formulas, the lower the number, the easier it is to read the document. For Flesch-Kincaid (Kincaid et al. 1975  Dale-Chall formula (DC) As mentioned before, the Dale-Chall formula (Dale and Chall 1948;Chall and Dale 1995) requires a 3,000 English words list which is familiar to U.S 4th grade (9-10 years old) students. As an alternative to 17 https://pypi.org/project/bnlp-toolkit/ 18 https://w.wiki/aRd  this, we manually annotate 3,396 Bengali words. According to the Dale-Chall formula, any word that is not in our Bengali 3,396 words list is treated as a difficult word. Table 3 indicates poor performance for Flesch reading ease, Flesch-Kincaid grade level, Gunning fog, and SMOG, therefore we can say that these formulas are not ideal for measuring the readability of the Bengali texts. On the other hand, the ARI performs relatively well as it correctly measures the age group of 8 documents out of 14 documents.

Supervised Neural Approaches
Due to the long-range dependencies of RNNs and the unavailability of large-scale human-annotated corpora, encoding of documents still lacks satisfactory solutions (Trinh et al. 2018). On the other hand, some document-level readability formulas, such as the SMOG index, require at least 30 sentences to measure a score. 18 In this section, we tackle these issues by dividing the document-level task into a supervised binary sentence classification problem where we have two classes, simple and complex. We design neural architectures to experiment with our sentence level dataset, which is presented in the dataset section.
Additional Feature Fusion The words present in complex sentences can be longer in terms of characters than the words in simple sentences. Therefore, we choose Character Length (CL), which indicates the total number of characters in a sentence, including white spaces as our additional readability feature. Moreover, the number of Consonant Conjunct (CC) is also an indicator for the complex sentence (Sinha and Basu 2016;Phani, Lahiri, and Biswas 2019). Unfortunately, these prior works did not present any specific algorithm to compute consonant conjunct for Bengali texts. Hence, we present a detailed procedure for counting consonant conjunct in Algorithm 1. To evaluate our consonant conjunct counting algorithm, we manually create a dataset with 341 words 19 and their corresponding consonant conjunct count. Our algorithm can successfully count the CCs present in all 341 words from our dataset. Figure 2 presents a visual representation of the counting of CC and CL for two example sentences.

Ablation Experiments
We select Bidirectional LSTM (BiLSTM) (Schuster and Paliwal 1997) and BiLSTM with attention mechanism (Raffel and Ellis 2016) as baseline models. As we can see from Table 4, BiLSTM model has achieved better performance from the baseline models, therefore, we extend the BiLSTM model by adding global average pooling and global max-pooling layers (Boureau, Ponce, and LeCun 2010). We use this model for our ablation experiments with the additional features to demonstrate the effects of CL and CC feature fusion. For each input sentence, we calculate the CL and CC to concatenate with the pooling layers. Given an input sentence s j , its word sequence w 1 , w 2 , w 3 ...w |sj | is fed into a word embedding layer to obtain embedding vectors x 1 , x 2 , x 3 ...x |sj | before passing it to the BiLSTM layer. The word embedding layer is parameterized by an E w ∈ IR K×|V | embedding matrix, where K is the embedding dimension and |V | is the vocabulary size. The overall process is summarized in Figure 3. We use all pre-trained language models available to date for the Bengali language, which includes:  • 300 dimensional Word2vec pre-trained model (Mikolov et al. 2013).
• 300 dimensional BPEmb (Heinzerling and Strube 2018) model, which is based on Byte-Pair encoding, provides a collection of pre-trained subword embedding models for 275 languages including Bengali.

Results
We present the detailed ablation experiment results of our test set in Table 4. We didn't perform any explicit prepossessing before computing the semantic representation using the embedding layer lookup in Figure 3. We compute accuracy, precision, recall, and F1 score to compare the performance of our readability prediction model with the baselines. We perform a detailed ablation study with the variations of pre-trained embedding models and our additional feature fusions. After applying all the models from Table 4 to our test set, we get maximum accuracy (86.7%) and F1 score (86.7%) from the combination of BiLSTM with pooling, CL, CC, and embeddings from LaBSE model. As shown in Table 4, the impact of additional features such as CL and CC of our readability prediction model is significant, achieving maximum Accuracy and F1 scores 6 times out of 8 cases.

Supervised Pretraining
We also experiment with fastText supervised text classification techniques (Joulin et al. 2017). These models are based on multinomial logistic regression. We build 3 classifiers using word n-grams (unigram, bigram, and trigram) and character n-grams (2 to 6 length) with learning rate = 0.5 and 50 epochs. For all 3 cases, we use hierarchical softmax (Peng et al. 2017) as the loss function for faster training. From Table 5, we observe that the model with word trigram and character n-grams achieves maximum Accuracy and F1 score.

Bengali Readability Analysis Tool
We design a Bengali readability analysis tool capable of providing in-depth information of readability and complexity of the documents which is shown in Figure 4. For an input document D with N sentences, our Bengali readability checker can highlight the simple and complex sentences. We calculate the readability score based on the ratio of the predicted simple sentences to the total number of sentences in the document D and assign different ratings based on the readability scores.
Since ARI performs well on the Bengali document-level readability analysis and it is resource independent, therefore, we use the scores returned from this formula to predict the agegroup of the input document. The example document D in Figure 4 will be reasonably easy to read for most people at least with age between 10-11.

Conclusion
In this paper, we present a readability analysis tool that would be useful for educators, content writers or editors, researchers, and readers of different ages. We adopt document-level readability formulas traditionally used for U.S. based education system to the Bengali education system with a proper age-toage comparison. Moreover, we divide the task into sentencelevel and design supervised neural models, which will serve as a baseline for the future works of this task. We present several human-annotated corpora, dictionaries, and an algorithm, which can be useful for several other tasks of this low-resource language. In the future, we wish to improve the quality of our system by increasing the size of our sentencelevel dataset and will present a user-study based on our tool. Also, we will focus on the readability analysis of Bengali-English code-mixed texts.