{'fraction_non_alphanumeric': 0.02610017737984627, 'fraction_numerical': 0.00506799560773714, 'mean_word_length': 4.646161182641869, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Naturalization of Text by the Insertion of Pauses and Filler Words

In this article, we introduce a set of methods to naturalize text based on natural human speech. Voice-based interactions provide a natural way of interfacing with electronic systems and are seeing a widespread adaptation of late. These computerized voices can be naturalized to some degree by inserting pauses and filler words at appropriate positions. The first proposed text transformation method uses the frequency of bigrams in the training data to make appropriate insertions in the input sentence. It uses a probability distribution to choose the insertions from a set of all possible insertions. This method is fast and can be included before a Text-To-Speech module. The second method uses a Recurrent Neural Network to predict the next word to be inserted. It confirms the insertions given by the bigram method. Additionally, the degree of naturalization can be controlled in both these methods. On the conduction of a blind survey, we conclude that the output of these text transformation methods is comparable to natural speech.


I. INTRODUCTION
Electronic systems that interact with humans are making their way into everyday life.
From robots in the house to personal assistants on our phones, these systems aid us in completing both simple and complex tasks. Interfacing with these electronic systems has also evolved. The focus has shifted to voice-based interactions over the more traditional text-based interactions seen in legacy systems 1 . Voice commands and feedback provide a more natural way for us to interface with these systems. However, even with the rapid development of the state of the art systems, we still largely interact with computerized voices. With this increase in voice-based interaction, we propose a set of methods to naturalize them. Naturalization of speech can be achieved in multiple ways. In this article, we transform text by adding pauses and filler words such as 'uh' and 'um' to an input sentence.
Google's Duplex 2 , can make calls on your behalf to book appointments or reserve tables and was unveiled in 2018. Despite being a computerized voice, pauses and filler words make the conversation more natural. Motivated by this observation, we propose the following methods. Moreover, these methods can be integrated as a module in a pipeline that takes a voice query as input and outputs a voice response. This module would take the response text from an information retrieval system as input and output naturalized text which can be input for a text-to-speech (TTS) module.
Previous cognitive studies of natural human speech indicate that the addition, deletion, modification of words, sentences of standard speech are observed in conversations 3 . These can be termed as disfluencies in speech. To achieve naturalization of speech, we resort to the addition of pauses and filler words such as 'uh' and 'um'. These pauses and filler words are an important part of natural conversations and unconsciously end up as a part of them.
In natural conversations, pauses and filler words are often used when a conversationalist is either emphasizing on a part of the conversation or gathering their thoughts. By inserting them in standard text, the text is transformed into a more natural text. An example Standard Text: Let's see, Susan is 15. Aundrea is 9. Every stupid cliche you hear about kids, they change your life, they make you a better person, they make you whole, it's all true! Now I get it.
Transformed Text: (um) Let's see, (pause) Susan is 15. Aundrea is 9. every stupid cliche you hear about kids, (pause) they change your life, they make you a better person, they make you whole, (pause) it's all true! Now I get it.

II. LITERATURE REVIEW
As observed, in a conversational context, the transformed text sounds more natural than the standard text. After studying the structure of conversations, we observe the frequency of these filler words in conversations 4 and the roles they play 5 . Previously, in 6 a language model is developed that predicts disfluencies probabilistically and uses an edited, fluent context to predict the following words. In 7 , a combination of prosodic cues modeled by decision trees and word-based event N-gram language models automatically detects disfluencies. These models detect disfluencies such as pauses and filler words in speech. In 8 , an "Editing Term" is inserted from the speech synthesis inventory and local prosodic modifications are applied at units adjacent to it. This suggests synthetic disfluent speech can be generated by making local changes. In 9 , an empirical text transformation method is proposed for spontaneous speech synthesizers which are trained on transcribed lecture monologue. The novelty of this article includes (a) using transcribed movie dialogues as training data, (b) a hybrid approach in inserting pauses and filler words to increase confidence, (c) tunable degree of naturalization, (d) blind survey to assess the quality of the transformed text.
First, we discuss data preprocessing followed by the two proposed methods. The bigram method involves the storing of bigrams with a pause or a filler word and its corresponding frequency in the training data. During transformation, a probabilistic approach is adopted to insert an appropriate pause or filler word. The hybrid approach first transforms text using the bigram method. A sequence to sequence model tuned to the context is then employed to validate these insertions.

III. DATA PREPROCESSING
The closest imitation of natural human speech which is well recorded and structured is in the form of dialogues from movie scripts. We use the Movie Dialogue Corpus from Kaggle 10 as well as the Baskin Engineering, Film Corpus 2.0 11 . Combined, this forms a corpus of over 2.2 × 10 6 lines of text. Data preprocessing (I) outputs the training data and the steps involved are 1. Extracting the lines containing disfluencies such as pauses and "uh", "um" 2. Care is taken to identify and extract multiple variations of the above disfluencies such as "uhh" and "uhm" 3. Two or more periods or dashes are also considered as pauses and are extracted forming an intermediate corpus 4. Identified pauses are replaced by "(pause)" and "uh", "um" are replaced by "(uh)", "(um)" to maintain uniformity in the final corpus This forms a corpus of over 2 × 10 5 lines and serves as the training data.

A. Bigrams
A sentence is taken as an input and the transformed sentence with the inserted pauses and filler words is output. Parameters that control the degree of naturalization can be set.
Roman numerals relate to steps in the Figures. Following are the steps of the Bigrams (II) method 1. A list of bigrams of words containing pauses and filler words as either the predecessor or the successor to a word is constructed from the training data. The frequency of these bigrams is recorded. Additionally, a special token is appended at the start of each sentence to account for the possibility of an insertion at the beginning of the sentence.
2. Clean the input sentence by dropping punctuation and numbers. The case is uniformly set to lower.
3. Find all possible insertions in the input sentence by splitting it into bigrams and comparing (III) the predecessor and successor with the list of bigrams from step 1. This is the draw set, D.
4. Draw a subset from D based on the degree of naturalization and a probability distribution. The degree of naturalization is said to be the percentage of inserted words given the length of the input sentence.

Degree of naturalization =
Number of inserted words Length of input sentence A probability distribution that fits the frequency of bigrams from step 1 is used to construct a subset of D. The set of probabilities, P with which a bigram appears in a subset of D is It is observed that some bigrams occur more frequently than others in a corpus of text.
This is a property of natural speech that we leverage to output more intelligible transformations using the probability distribution mentioned in Equation 2.
Step 1 can be performed independently and pre-computed before the method sees an input sentence. Therefore, this step does not contribute to the overall speed of the transformation. Table I lists the five most frequently occurring bigrams from the training data.  2. Instantiate a tokenizer. The tokenizer converts the input data text into sequences.
The sequence length is set to three. The method uses the context of the two preceding words to predict one word.
3. The sequences are converted to categorical data and an RNN is trained using parameters defined in Table II. Hybrid predicted next word: (pause)

Correct Prediction
Intermediate output: Let us (pause) try this one more time We observe that the model confirms a pause more frequently than the other filler words.
According to 3 , pauses in natural language can be replaced by other filler words. Longer sentences are also more likely to begin with these filler words. Therefore, we take the liberty to replace the first n pauses by the other filler words where n is based on the degree of naturalization. Figure 1

V. RESULTS AND DISCUSSION
As there are multiple ways of achieving natural speech, we aim to transform the text into the most natural-sounding version. To illustrate this, following is an example Variation 1: This is (pause) the first time I am (uh) coming here.
Variation 2: This is the (uh) first time (pause) I am coming here.
Variation 3: This (uh) is the first time I am coming (pause) here.
Therefore, we use a medium-sized pool of 275 odd individuals to evaluate the outputs and determine if they sound natural or not. We refer to them as respondents hereafter. The outputs are generated by both, the bigram and the hybrid method. Degree of naturalization is 9.5% for both methods. While answering the question -'Which of the following sounds more natural?', judging between two text excerpts is more difficult than judging between two voice clips of the same text excerpts. Hence, the respondents were presented with two voice clips. One was a performance of an original interview's transcript and the other was a performance of the transformed text of the same interview's transcript. Both were performed by the same voice actor. Here is an example of the output text from the bigram method followed by an example from the hybrid method    context increase, the methods get more robust and consequently, the transformations more natural.
As stated previously, our methods of text transformation are fast. This is to ensure that this text transformation module can be integrated before a TTS module. Table V details the timing data in seconds for both methods and their averages. All runs are performed with equal computing power to maintain uniformity. All samples are long sentences and the degree of naturalization is 9.5% for both methods. Sentence length and degree of naturalization are directly proportional to the time taken to generate an output sentence. We observe that the bigram method is multiple factors faster than the hybrid method.

VI. CONCLUSION
In this article, we propose a novel, fast transformation method that has tunable naturalization. It also uses a hybrid approach to confirm the transformations. We evaluate our methods using a medium-sized pool of individuals to evaluate the outputs and determine if they are natural. We conclude that our methods yield near-natural transformations of text and can be integrated as a module into a voice query to voice response pipeline.
Acknowledgements: The authors would like to thank PES University and Dr. S S Shylaja, Chairperson, Department of Computer Science and Engineering, PES University for their support.

Conflicts of Interests:
The author declares no conflict of interest