{'fraction_non_alphanumeric': 0.029692146230540492, 'fraction_numerical': 0.01928458981983558, 'mean_word_length': 5.067657203502256, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 5, 'lorem ipsum': 1, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

The T5 model and its unified text-to-text paradigm contributed in advancing the state-of-the-art for many natural language processing tasks. While some multilingual variants of the T5 model have recently been introduced, their performances were found to provide suboptimal performances for languages other than English if compared to monolingual variants. We are motivated by these findings to introduce IT5, the first family of encoder-decoder transformer models pretrained specifically on Italian. We perform a thorough cleaning of a web-crawled Italian corpus including more than 40 billion words and use it to pretrain three IT5 models of different sizes. The performance of IT5 models and their multilingual counterparts is then evaluated on a broad range of natural language understanding and generation benchmarks for Italian. We find the monolingual IT5 models to provide the best scale-to-performance ratio across tested models, consistently outperforming their multilingual counterparts and setting a new state-of-the-art for most Italian conditional language generation tasks.


Introduction
The text-to-text paradigm introduced by the T5 model (Raffel et al., 2020) has recently been widely adopted as a simple yet powerful generic transfer learning approach for most language processing tasks (Sanh et al., 2021;Aribandi et al., 2022). Although the original T5 model was trained exclusively on English data, the same architecture has been extended to a massively multilingual setting covering more than 100 languages by mT5 and ByT5 (Xue et al., 2021a,b), following recent advances in the multilingual pre-training of large language models such as mBERT, XLM, XLM-R and mDeBERTa (Devlin et al., 2019;Conneau and Lample, 2019;Conneau et al., 2020;He et al., 2021). Multilingual language models were shown to excel in cross-lingual and low-resource scenarios, but multiple studies have highlighted their suboptimal scale-to-performance ratio if compared to monolingual counterparts for language-specific applications in which data are abundant (see Nozza et al. 2020 andRust et al. 2021 for an overview).
Following this line of research, Nagoudi et al. (2021) pre-trained several T5 models on the Arabic language, obtaining an improvement in performances from the multilingual mT5 model (and other transformer variants) on Arabic conditional language generation tasks such as news summarization, headline generation and question generation. Similarly, Carmo et al. (2020) pre-trained a Portuguese T5 model and showed competitive performances on named entity recognition and natural language inference.
In this work, we follow an approach similar to the one of Nagoudi et al. (2021) to pre-train and evaluate a set of three Italian T5 models of different sizes, which we identify under the name IT5. In Section 3 we present the procedure we adopted to clean the Italian mC4 corpus (Xue et al., 2021b) used for pre-training the IT5 models, and we briefly cover the set of pre-training parameters. Section 4 describes the multilingual baselines and the downstream tasks we used to evaluate the performances of fine-tuned IT5 models and presents the obtained results. Finally, we discuss our findings and future directions in Section 5. To the best of our knowledge, the IT5 models are the first publicly available encoder-decoder models pre-trained exclusively on the Italian language, and we view them as a substantial contribution towards the advancement of Italian NLP. Overall, our contributions are the following: • We introduce a thoroughly cleaned version of the Italian mC4 corpus and use it pre-train three IT5 models of various dimensions.
• We evaluate the IT5 models on a broad set of natural language understanding and generation tasks for Italian, showing consistent improvements compared to multilingual models and the current state-of-the-art.
• We publicly release all the code, data, pretrained and fine-tuned checkpoints for further experimentation by the research community.

Text-to-text Transfer Transformers
The Text-to-text Transfer Transformer (T5) model introduced by Raffel et al. (2020) adapts the original Transformer architecture proposed by Vaswani et al. (2017) to a scenario where multiple natural language processing tasks are first reformulated into a unified text-to-text format and then used alongside masked span prediction for semisupervised pre-training. The encoder-decoder architecture of T5 is especially suited for sequenceto-sequence tasks (Sutskever et al., 2014), which cannot be performed by encoder-only models like BERT (Devlin et al., 2019) and can prove to be challenging for decoder-only models like GPTs (Radford et al., 2019;Brown et al., 2020) due to the lack of explicit conditioning on source context. The same architecture can be easily extended to most natural language understanding tasks by converting them to the text-to-text format, making the T5 model highly versatile to most NLP settings.

Pre-trained Language Models for Italian
The rapid progress achieved in recent years by the NLP research community on shared benchmarks such as SuperGLUE (Wang et al., 2019) is primarily driven by model scaling and the large-scale pre-training, two components requiring both high technical expertise and heavy computational resources. These strict conditions significantly contributed to exacerbating inequalities in the availability of state-of-the-art systems across languages other than English. This is especially true in the case of the Italian language, which despite not being "low-resource" under any possible definition of the term, can currently count on a minimal set of publicly available pre-trained language models (Miaschi et al., 2021), and most notably on a single pre-trained model for text generation (GePpeTto by De Mattei et al. 2020a, built upon the GPT-2 small model architecture).

Data and Model Pretraining
The original T5 model is pre-trained on the Colossal Clean Crawled Corpus (C4) (Raffel et al., 2020), a large collection including roughly 750GB of webscraped English texts sourced from the Common-Crawl. This was subsequently cleaned employing heuristics aimed at removing templated fillers, text deduplication, Javascript code, slurs and non-English texts. The multilingual counterpart of T5 adopts a similar procedure to create mC4 (Xue et al., 2021b), a multilingual version of the C4 corpus including 107 languages. While the authors of mC4 still perform text deduplication, language detection and bad words removal, they lower the language detection threshold to 70% and omit other useful heuristics due to the great variability across the covered character systems, such as filtering sentences having non-standard end-of-sentence punctuation. As a consequence, the resulting corpus has an overall lower quality, with a recent study finding 16% of examples in a random sample of mC4 being associated to the wrong language tag, and 11% of them not containing any linguistic information (Kreutzer et al., 2022).
In light of this, we decided to perform a more thorough data cleaning of the Italian portion of the mC4 before pre-training our IT5 models, as described in the following section.

Cleaning the Italian mC4 Corpus
The original Italian portion of the mC4 Corpus includes 1024 training files in JSON format of roughly 220MB each and eight validation files of roughly 24MB each, for a total of approximately 215GB of raw text, making it the most extensive publicly available Italian corpus to this date. We use a public implementation 1 reproducing and augmenting the original C4 data cleaning pipeline to perform sentence tokenization inside every document and remove sentences containing: • words from a manually-selected subset of the Italian and English List of Dirty Naughty Obscene and Otherwise Bad Words; 2 • less than three words, or a word longer than 1000 characters; • an end symbol not matching standard end-ofsentence punctuation for Italian; • strings associated to Javascript code, lorem ipsum, English and Italian privacy policy/cookie disclaimers.
After filtering sentences and recomposing documents from the remaining sentences, we further exclude documents containing less than five sentences or containing either less than 500 or more than 50,000 characters or not being identified as prevalently Italian by the LangDetect library. 3 The cleaning process was performed by using a parallel processing pipeline on a TPUv3-8 virtual machine with 96 CPU cores on Google Cloud Platform and required roughly 10 hours of computation due to the demanding steps of sentence tokenization and language detection. The size of the resulting corpus, which we name Clean Italian mC4 Corpus 4 , is roughly halved by this cleaning procedure and includes approximately 215GB of raw text in Italian. This corresponds to a rough estimate of 103 million documents or 41 billion words.

Model and Training Parameters
Having obtained a clean corpus for pre-training, we train a SentencePiece unigram subword tokenizer (Kudo, 2018) with a vocabulary size of 32'000 words on the first 10 million cleaned documents and we use it to pre-train three models following the canonical small, base and large sizes defined in Raffel et al. (2020). Table 1 provides a brief overview of size-specific parameters, including training times and the number of steps, while Appendix B provides more comprehensive information about the overall architecture. Models are trained on a TPU v3-8 accelerator on Google Cloud Platform using the JAX framework (Bradbury et al., 2018) and Huggingface Transformers (Wolf et al., 2020). We adopt the T5 v1.1 architecture 5 also used by the mT5 model, improving upon the original T5 by using GeGLU nonlinearities (Shazeer, 2020), scaling model hidden size alongside feedforward layers and pre-training only on unlabeled data, without dropout. All models are pre-trained with a learning rate of 5e-3 and a maximum sequence length of 512 tokens using the Adafactor optimizer (Shazeer and Stern, 2018) Figure 1 shows the computed loss during the training process for the three models. We used the Google Cloud Carbon Footprint tool to estimate the overall amount of CO2 generated by the pre-training process and found it to be approximately equal to 7kgCO2, corresponding approximately to the emissions of a 60km car ride. 6

Fine-tuning Tasks
The following paragraphs provide an overview of the seven tasks we selected to evaluate the downstream performances of fine-tuned IT5 and mT5 models. Given the limited availability of evaluation resources for natural language generation in Italian, our selection aims to provide a comprehensive overview of canonical areas of the conditional text generation paradigm such as summarization, style transfer and question generation. Moreover, we also include a question answering task to provide a direct comparison of IT5 performances against encoder-based extractive systems.
Wikipedia Summarization To evaluate summarization capabilities in the encyclopedic domain, we use the Wikipedia for Italian Text Summarization (WITS) corpus (Casola and Lavelli, 2022), containing roughly 700'000 articles extracted from a cleaned dump of the Italian Wikipedia alongside their leading sections, that are treated as summaries for the summarization task. We mimick the same evaluation setup as the original authors by using a test set of 10'000 examples.

News Summarization
We evaluate the news article summarization setting by concatenating two collections of articles from the Fanpage.it and Il-Post newspapers scraped by the Applied Recognition Technology Laboratory 7 and published on the Huggingface Datasets Hub (Lhoest et al., 2021) alongside monolingual and multilingual modeling baselines. We refer to this concatenated corpus as NewsSum-IT from here onwards. We fine-tune our systems on the whole training set, including more than 100'000 articles and respective short summaries, and evaluate them separately on the two test sets defined by the dataset creators. We report the averaged metrics across the two newspapers in the results section.
Question Answering We evaluate our models on extractive question answering using the SQuAD-IT dataset (Croce et al., 2018), which consists of a subset of roughly 50'000 paragraph-questionanswers triplets produced by automatically translating the original SQuAD dataset (Rajpurkar et al., 2016) and filtering out problematic examples. For using the dataset with our models and baselines, we frame the QA task as a sequence-to-sequence problem aimed at generating plausible responses given a source text in the format <PARAGRAPH> Domanda: <QUESTION>. We use the canonical splits for training and testing and evaluate generated answers using the script provided by the original authors.
Question Generation We leverage the same dataset used for QA to evaluate question generation capabilities by reversing the text triplets, making the model predict a plausible question given a source text in the format <PARAGRAPH> Risposta: <ANSWER>, where the answer is the first among the available answers for the given example. We adopt the same splits used for QA.
News Headline Style Transfer We evaluate style transfer abilities in the news domain on the 7 https://huggingface.co/ARTeLab CHANGE-IT shared task (De Mattei et al., 2020b), which includes two sets of roughly 60'000 newspaper articles and headlines each extracted from the left-leaning Italian newspaper la Repubblica and the right-leaning Il Giornale, respectively. We train and validate our models on the canonical splits using the same cross-source article-to-headline generation approach used for the original baselines by the authors since we find it to work more effectively in the absence of high-quality article alignments across all examples of the two corpora. We report the averaged performances for the two style transfer directions (Il Giornale ↔ la Repubblica).
News Headline Generation We combine the two CHANGE-IT subsets to create a corpus of roughly 120'000 news articles and headlines pairs, which we refer to with the name HeadGen-IT. We use HeadGen-IT to evaluate news headline generation performances of our models, using the same test splits defined in CHANGE-IT for prediction.
Formality Style Transfer We evaluate the formality style transfer capabilities of our models on the Italian subset of the XFORMAL dataset (Briakou et al., 2021), containing a training set of 115'000 forum messages automatically translated from the GYAFC corpus (Rao and Tetreault, 2018) and covering the topics of entertainment, music, family and relationships, and a small test set of 1000 formal-informal pairs obtained directly in Italian from four crowdworkers via Amazon Mechanical Turk. We evaluate our models in both style transfer directions (Formal ↔ Informal).

Evaluation Metrics
We adopt a common combination of text-based and trained metrics as a default for evaluating model performances in most settings. We use the language-independent ROUGE metric (Lin, 2004) in its unigram (R1), bigram (R2) and Longest Common Subsequence (RL) variants to evaluate lexical matches, and the trained BERTScore metric (Zhang et al. 2020;BS) to evaluate correspondence at the semantic level. Since BERTScore requires a pre-trained BERT-like model for the target language, we use a popular BERT model pre-trained on Italian for the evaluation. Following the authors' recommendation 11 , we compute and use baseline scores (see Appendix C for details) in order to  broaden the range of the metric and remove noise. For the QA task, we instead use the exact-match (EM) and F1-score (F1) metrics routinely used for evaluation. Finally, for the news headline style transfer task, we use trained classifiers provided by the authors 12 to ensure headline-headline (HH) and headline-article (HA) coherence.

Baselines
For every selected task, we compare the performances of the three IT5 models to those previously reported in the literature. Moreover, we adopt the same fine-tuning procedure for training and evaluating two sizes (small and base) of the multilingual T5 model (mT5) (Xue et al., 2021b). This allows us to assess the validity of our pre-training procedure and to observe whether the monolingual setting yields a performance improvement. We em-12 https://github.com/michelecafagna26/ CHANGE-IT 8 https://huggingface. co/antoniocappiello/ bert-base-italian-uncased-squad-it 9 https://huggingface.co/ARTeLab/ mbart-summarization-ilpost 10 https://huggingface.co/ARTeLab/ mbart-summarization-fanpage phasize that mT5 models are considerably larger than regular T5 architectures due to a larger vocabulary and embedding matrix. While this fact should not affect their computational efficiency, their size in memory could limit their utilization by most NLP practitioners. Additional details on the parametrization across fine-tuning tasks for mT5 and IT5 models are provided in Appendix D.

Results and Discussion
Tables 2, 3 and 4 present the results of our finetuning experiments. Given the broad scope of our analysis, we limit ourselves to comment salient trends we observe across tasks.
IT5 models provide state-of-the-art performances for language generation and understanding tasks in Italian. The IT5 models outperform multilingual models and previous systems in 6 out of 8 evaluated tasks, with noticeable improvements over mT5 systems in particular for question answering and generation and for headline-headline coherence on the news headline style transfer task. More specifically to the QA setting, the IT5 Large model establishes a new top score for exact matches, outperforming most extrac-  Table 3: Performance of IT5 models and mT5 models on Italian formality style transfer (XFORMAL IT) for the formal-to-informal (F → I) and informal-to-formal (I → F) directions and on the news headline generation task (HeadGen-IT). Best scores are marked in bold.  tive systems and closing the gap with the XLM-R Large system adopted by Riabi et al. (2021), although the latter is trained on more data using a specific synthetic data augmentation procedure.

CHANGE-IT
Multilingual models can still be helpful in specific applications and when using translated data. We observe that multilingual language models are still obtaining the best performances in the news summarization and the formal-to-informal style transfer tasks. In the case of news summarization, we attribute the performance gap in large part to the scale of the mBART baseline model. For the formality style transfer task, instead, after a minimal error analysis, we conjecture that translation errors and English acronyms present in the noisy training split of XFORMAL act as outof-distribution samples in the monolingual setting, disrupting the performances of IT5 systems, but are captured more easily by multilingual systems which were exposed by multiple data distributions by design. This would indicate a better fit of multilingual pre-trained models for such settings if verified. We leave a more thorough analysis assessing the systematicity of these patterns to future work.
Scaling model size does not guarantee an increase in performance if not supported by an increase in computational resources. Contrary to the findings of "scaling laws" (Brown et al., 2020) related to model size for pre-trained Transformer models, we do not observe a systematic increase in downstream performances for base and large IT5 models across all tasks when compared to their small counterparts, despite reaching lower loss values and higher pre-training accuracies. While recent work highlighted how better pre-training performances do not always correspond to better downstream scores for T5 models (Tay et al., 2022), we hypothesize that in our case, the lack of improvement is related mainly to insufficient computational resources that limited the maximal batch size during training to 128, as opposed to the 2048 figure reported in the original work by Raffel et al. (2020).

Conclusion
This paper introduced IT5, the first family of largescale encoder-decoder models pre-trained on the Italian language. We presented a detailed overview of the pre-training procedure, including the previous data cleaning steps, and performed a thorough evaluation of the IT5 models and their multilingual mT5 counterparts on a broad set of conditional language generation tasks for the Italian language. We obtained new state-of-the-art results across most evaluated tasks and concluded by discussing the shortcomings of large-scale monolingual language modeling when dealing with automatically translated data and limited computational resources. In light of our results, we find the study of time and computation trade-offs between the pretraining of monolingual models and a languagespecific continued pre-training of multilingual models promising to close the language gap in the performances of NLP models.
Despite our thorough cleaning procedure aimed at removing vulgarity and profanity, it must be acknowledged that models trained on web-scraped contents such as IT5 will inevitably reflect and amplify biases present in Internet blog articles and comments, resulting in potentially harmful content such as racial or gender stereotypes and conspiracist views. In light of this, the study of such biases is explicitly encouraged, and model usage should ideally be restricted to research-oriented and nonuser-facing endeavors.

Acknowledgments
We thank the Google TPU Research Cloud program for providing us with free access to TPU v3-8 machines used in pre-training the IT5 models and the Center for Information Technology of the University of Groningen for providing access to the Peregrine high performance computing cluster used in fine-tuning and evaluation experiments. We also thank the Huggingface team for creating the original template for TPU-compatible pre-training and fine-tuning scripts for T5 models and making them available during the Huggingface JAX/Flax Community Week.

References
Vamsi Aribandi, Yi Tay Table  7: Baseline scores for using dbmdz/bert-base-italian-xxl-uncased with the BERTScore evaluation framework. Table 6 contains task-specific parameters that were used for the fine-tuning experiments. For mT5 Small, IT5 Small and IT5 Base models we use a learning rate of 5e-4 and a batch size of 64 examples, while larger models (mT5 Base and IT5 Large) were fine-tuned with a leaning rate of 5e-5 and a batch size of 32. All models are fine-tuned with linear schedule with no warmup using the AdamW optimizer (Loshchilov and Hutter, 2019). We highlight that the batch sizes used for finetuning are significantly smaller from the canonical batch size of 128 adopted by Raffel et al. (2020) due to hardware limitations.