{'fraction_non_alphanumeric': 0.04154182524900802, 'fraction_numerical': 0.009474451372580775, 'mean_word_length': 4.54806828391734, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Unified Transformer-based Framework for Duplex Text Normalization

Text normalization (TN) and inverse text normalization (ITN) are essential preprocessing and postprocessing steps for text-to-speech synthesis and automatic speech recognition, respectively. Many methods have been proposed for either TN or ITN, ranging from weighted finite-state transducers to neural networks. Despite their impressive performance, these methods aim to tackle only one of the two tasks but not both. As a result, in a complete spoken dialog system, two separate models for TN and ITN need to be built. This heterogeneity increases the technical complexity of the system, which in turn increases the cost of maintenance in a production setting. Motivated by this observation, we propose a unified framework for building a single neural duplex system that can simultaneously handle TN and ITN. Combined with a simple but effective data augmentation method, our systems achieve state-of-the-art results on the Google TN dataset for English and Russian. They can also reach over 95% sentence-level accuracy on an internal English TN dataset without any additional fine-tuning. In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora for German and report the performance of our systems on the dataset. Overall, experimental results demonstrate the proposed duplex text normalization framework is highly effective and applicable to a range of domains and languages


INTRODUCTION
Text normalization (TN) is the process of converting written text to its spoken form. For example, 72 people were found should be verbalized as seventy two people were found (Fig.  1). TN is usually the first preprocessing step in a text-tospeech (TTS) system [1]. On the other hand, inverse text normalization (ITN) refers to the inverse process, which transforms spoken-domain text into its written form [2]. ITN is typically an important post-processing step of an automatic speech recognition (ASR) system. A challenge in TN/ITN is 1 The German dataset will be released upon the publication of this work 2 We have integrated our framework into NeMo, Nvidia's open-source Conversational AI toolkit https://github.com/NVIDIA/NeMo. the variety of semiotic classes [3,4]. Semiotic class denotes things like numbers, dates, times, etc., whose written forms are typically different from spoken forms. Another challenge is that there is low tolerance towards unrecoverable errors in a production setting. In the context of TN, acceptable errors are less severe errors that typically involve picking the wrong form of a word while otherwise preserving the original meaning (e.g., 35 mins → thirty five minute). In contrast, unrecoverable errors are errors where a totally different meaning is being conveyed (e.g., 35 mins → forty five minutes).
Traditional approaches to TN/ITN typically use handwritten grammars in the form of weighted finite-state transducers (WFST) [5,6,7] to handle semiotic classes such as date (e.g., May 24 ↔ May twenty fourth) or numbers (e.g., 72 ↔ seventy two). While WFST-based approaches work reasonably well and have previously been adopted by several production systems [6,4,8], they are expensive to scale across different languages. Due to the long tail of special cases, constructing WFST-based grammars typically requires extensive linguistic knowledge and manual effort to design handcrafted rules.
With the rise of neural networks (NN) in natural language processing (NLP) [9,10,11], several deep learning models have recently been introduced for either TN or ITN [4,12,1,13]. For example, [14] uses recurrent NN-s to learn a TN function from a large corpus of written text aligned to its spoken form. [15] explores the use of convolutional NN-s for TN. In [8], the author uses a sequence-to-sequence (seq2seq) model for TN with byte pair encoding (BPE) as sub-word units. More recently, [2] investigates the use of Transformerbased models for ITN. While these neural-based methods Overview of our duplex text normalization framework. In this example, the input to our system is a written text, and it needs to perform TN. Therefore, the task indicator is set to be TN.
can achieve impressive performance, they are not designed to work with TN and ITN simultaneously. As a result, to build a complete spoken dialog system with both TTS and ASR capabilities, two separate models for TN and ITN need to be implemented. This heterogeneity increases the technical complexity of the system, which in turn increases the cost of maintenance in a production setting.
In this work, we pursue an ambitious goal of addressing TN and ITN jointly. We propose a unified framework for building a single duplex system that can simultaneously handle the two tasks. Combined with a simple but effective data augmentation method, our Transformer-based systems achieve impressive results on both public and internal datasets. When trained on the public Google TN dataset [16], our systems achieve state-of-the-art results for both English and Russian. They can also reach over 95% sentence-level accuracy on an internal English TN dataset without any additional fine-tuning. In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora [17] for German and report the performance of our systems on the dataset. Overall, our proposed framework is highly effective and applicable to a range of domains and languages.

METHODS
Inspired by previous work [4], our framework consists of two main components ( Figure 2). Given an input sentence, a Transformer-based tagger is first used to identify all the semiotic spans in the input (i.e., numbers, times, dates, monetary amounts, etc.) (Section 2.1). After that, a Transformer-based The task indicator.

{B,I}-SAME
A span that should be kept the same.

{B,I}-PUNCT
A punctuation. {B,I}-TRANSFORM A semiotic span Table 1. The label set that the tagger uses. The prefix Bindicates the beginning. Any other token after the first is given the prefix I-.
normalizer is used to convert the semiotic spans into their appropriate forms (Section 2.2). For TN/ITN, typically, most of the tokens in the input can be kept the same. Tokens that need to be transformed belong to a small set of semiotic classes (e.g., measure, money, cardinal number, date, or time) [3]. Because of the tagger, the seq2seq normalizer only needs to work with few input spans. The normalizer does not have to transform the entire original input.
Our framework unifies TN and ITN by using taskindicating prefixes. To allow duplex mode, we append a task indicator prefix to each input to indicate whether the input is for TN or ITN [18]. This approach is conceptually simple, easy to implement, and effective. A single duplex system is also easier to maintain than two separate systems (one for TN, one for ITN).

Transformer-Based Tagger
Given an original input sequence T = (t 1 , ..., t n ) consisting of n tokens, we first append a task indicator token t 0 to the be-ginning of the sequence to indicate whether the model needs to do TN or ITN (i.e., t 0 ∈ {TN, ITN}). Therefore, the actual input sequence to our tagger is (t 0 , t 1 , ..., t n ). The role of the tagger is to predict a sequence of labels (y 0 , y 1 , ..., y n ), where y i is the label corresponding to token t i . Table 1 describes the label set that the tagger uses.
Our tagger first forms a contextualized representation for each input token using a Transformer encoder such as BERT [19] or RoBERTa [20]. Let X = (x 0 , ..., x n ) be the output of the Transformer encoder, where x i ∈ R d . We then feed the representations into a softmax layer to classify over the tagging labels (Table 1): where W ∈ R 8×d and b ∈ R 8 are trainable parameters. o i ∈ R 8 is a vector containing the predicted logits for token t i . To train the tagger, we use the cross-entropy loss function.

Transformer-Based Normalizer
Let S = {s 1 , ..., s m } be the set of all (predicted) semiotic spans in the input sequence T . Here, m denotes the number of semiotic spans. The role of the normalizer is to transform each semiotic span into its appropriate form (e.g., its spoken form if T is a piece of written text and the task is TN). For semiotic span s i , the actual input to the normalizer includes the task indicator, the left context of s i , the textual content of s i , and the right context of s i (Figure 2). We use two special tokens (denoted as <m> and </m> in Figure 2) to separate each semiotic span from its context. Since the surrounding context of a semiotic span may contain another semiotic span, the two special tokens are needed to highlight the span that the normalizer needs to pay the most attention to. Our normalizer model is based on the standard encoderdecoder Transformer architecture [21]. First, an input sequence of tokens is mapped into a sequence of input embeddings, which is then passed into the encoder. The encoder consists of a stack of Transformer layers that map the sequence of input embeddings into a sequence of feature vectors. The decoder is also Transformer-based. It produces an output sequence in auto-regressive manner: at each output time-step, the decoder attends to the encoder's output sequence and to its previous outputs to predict the next output token. The normalizer model is trained using standard maximum likelihood, i.e., using teacher forcing [22] and a crossentropy loss.
To make the normalizer more robust against the tagger's potential errors, we train the normalizer with not only correct semiotic spans but also with some other more "noisy" spans (Fig. 3). For example, let's consider the sentence "remind me at 4 pm today please". In addition to the semiotic span "4 pm" (and its context), we also use other spans such as "at 4 pm today" (and their contexts) as training examples for the normalizer (for this augmented case, the target output should

Component
Language Model English Tagger distilroberta-base English Normalizer t5-base Russian Tagger cointegrated/rubert-tiny Russian Normalizer cointegrated/rut5-base German Tagger bert-base-german-cased German Normalizer google/mt5-base Table 2. The pretrained language models we use in this work. The models are referred by their names in HuggingFace's Transformers library.
be "at four p m today"). This way even if the tagger makes some errors, there will still be some chance that the final output is still correct.

EXPERIMENTS AND RESULTS
Data and Experimental Setup For English and Russian, we use the standard Google TN dataset for training [16]. For German, we create a cleaned dataset from the Spoken Wikipedia Corpora [17]. We use pretrained Transformer-based language models from HuggingFace's library 3 . For English, we use a distilled version of RoBERTa for the tagger [20] and T5base for the normalizer [18]. For Russian, we use a distilled version of the multilingual version of BERT for the tagger [19], and a distilled version of the multilingual mT5 for the normalizer [24]. For German, we use a German version of BERT for the tagger [19], and we use the multilingual mT5 for the normalizer [24]. Table 2 summarizes the list of language models we use. The TN framework is open-sourced in NeMo 4 , where one can find an implementation details and training hyper-parameters.
Comparison with Previous Methods. Table 3 summarizes the performance of our systems and compares them to other baselines. The duplex systems trained with data augmentation outperform the baselines on all languages. Furthermore, the duplex systems achieve results comparable to   Table 5. Some of the unrecoverable TN errors. unrecoverable errors, seven are related to URLs, three are related to numbers, and one is miscellaneous (Table 5).

CONCLUSION
This paper introduces a novel unified framework for building duplex systems that can simultaneously handle both direct and and inverse text normalization. Experimental results on both public and internal datasets demonstrate the effectiveness of our framework. Our best systems achieve state-of-the art results on the Google TN dataset. An interesting future direction is to investigate semi-supervised learning techniques to reduce the amount of data required for training our systems. Another direction is to build a single multilingual duplex system that simultaneously handles multiple languages.