{'fraction_non_alphanumeric': 0.03809800785226116, 'fraction_numerical': 0.017594881489021375, 'mean_word_length': 5.140625, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?

Text summarization is a challenging task within natural language processing that involves text generation from lengthy input sequences. While this task has been widely studied in English, there is very limited research on summarization for Vietnamese text. In this paper, we investigate the robustness of transformer-based encoder-decoder architectures for Vietnamese abstractive summarization. Leveraging transfer learning and self-supervised learning, we validate the performance of the methods on two Vietnamese datasets.


Introduction
In recent years, Transformer-based architecture models and pre-trained language models (LMs) have played an important role in the development of Natural Language Processing (NLP) systems. These large pre-trained models such as ELMo (Peters et al., 2018), GPT (Brown et al., 2020), BERT (Devlin et al., 2018) are trained on a large corpus and have the ability to derive contextual representation of the language(s) in the training data. After pre-training is complete, these models have achieved state-of-the-art results on a broad range of downstream tasks (Devlin et al., 2018). These self-supervised learning methods make use of learning objectives such as Masked Language Modeling (MLM) (Devlin et al., 2018) where random tokens in the input sequence are masked and the model attempts to predict the original tokens in order to gain a better understanding of context. The successes of pre-trained models in English have inspired new research efforts to development pretrained models in languages such as Vietnamese (i.e., PhoBERT (Nguyen and Nguyen, 2020) and ViBERT(Bui et al., 2020)). There are also ongoing efforts to develop multilingual pre-trained models (mT5 (Xue et al., 2020), mBART (Liu et al., 2020)) in order to improve performance across multiple languages by learning both general and languagespecific representations.
Text summarization is a summarization evaluation task in which an input will be a free-form text paragraph or document(s) and the output sequence will be a short summarization of the input. Although there has been extensive study in plain English text summarization (Manor and Li, 2019) and cross-lingual tasks (Zagar and Robnik-Sikonja, 2020;Xu et al., 2020), there have been minimal studies into the Vietnamese text summarization.
We will perform robust experiments on Transformer-based encoder-decoder models applied to Vietnamese Summarization Tasks. Following the transformer architecture from (Vaswani et al., 2017) and the Bert2Bert architecture (Rothe et al., 2019), we will use existing monolingual Vietnamese pre-trained models and multilingual pretrained models to perform our experiments on the Vietnamese Summarization task. We will finetune the models on two datasets: Wikilingual (Ladhak et al., 2020) and Vietnews (Nguyen et al., 2019). Our results shows that mT5 and mBART achieve more competitive results than monolingual Vietnamese pretrained model. This indicates that there are significant opportunities to further improve performance on Vietnamese summarization tasks (and Vietnamese NLP in general) with the development of stronger mongolingual pre-trained models.

Related works
There are many abstractive summarization studies in English. In an early example, (Gehrmann et al., 2018) employed a bottom-up content selector (Bot-tomUp) to determine which phrases in the source document should be part of the summary, and then a copy mechanism was applied only to pre-selected phrases during decoding. Their experiments obtained significant improvements on ROUGE for some canonical summarization datasets.
In recent years, pre-trained language models have been used to enhance performance on language generation tasks. (Liu and Lapata, 2019) developed a Transformer-based encoder-decoder model so pre-trained language models like BERT can be adopted for abstractive summarization task.
Here, the authors proposed a novel document-level BERT-based encoder (BERTSum) and a general framework encompassing both extractive and abstractive summarization tasks. Based on BERTSum, Dou et al. (2021) introduced GSum that effectively used different types of guidance signals as input in order to generate more suitable words and more accurate summaries. This model accomplished state-of-the-art performance on four popular English summarization datasets. Meanwhile, there are a small number of studies on Vietnamese text summarization. Most of these focus on inspecting extractive summarization. The researchers (Nguyen et al., 2018) compared a wide range of extractive methods, including unsupervised ranking methods (e.g., LexRank, LSA, KL-divergence), supervised learning methods using TF-IDF and classifiers (e.g., Support Vector Machine, AdaBoost, Learning-2-rank), and deep learning methods (e.g., Convolutional Neural Network, Long-Short Term Memory). Similarly, the authors (Nguyen et al., 2019) also evaluated the extractive methods on their own dataset, which was released publicly as a benchmark for future studies.
Recent work (Quoc et al., 2021) investigated the combination of pre-trained BERT model and an unsupervised K-means clustering algorithm on extractive text summarization. The authors utilized multilingual and monolingual BERT models to encode sentence-level contextual information and then ranked this information using K-means algorithm. Their report showed that monolingual models achieved better results compared when to multilingual models performing the same extractive summarization tasks. However, due to the lack of studies on Vietnamese abstract summarization, we compare both multilingual and monolingual encoder-decoder models.

BERT
BERT (Devlin et al., 2018) (Bidirectional Encoder Representations from Transformers) is a multilayer bidirectional Transformer encoder based on the architecture described in (Vaswani et al., 2017). In BERT, each input sequence is represented by a sum of the token embeddings, segment embeddings, its positional embeddings. BERT uses the WordPiece tokenizer to split words into subwords.

PhoBERT
PhoBERT (Nguyen and Nguyen, 2020) is the first public large-scale mongolingual language model pre-trained for Vietnamese. PhoBERT follows the architecture proposed by . The PhoBERT pretraining approach is also based on RoBERTa , employing the implementation in fairseq  which optimizes the BERT pre-training procedure for more robust performance. The model is trained on a 20GB word-level Vietnamese news corpus with maximum sequence length of 256.

ViBERT
ViBERT (Bui et al., 2020) leverages checkpoints from the multilingual pre-trained model mBERT (Devlin et al., 2018) and continues training on monolingual Vietnamese corpus (10GB of syllablelevel text). The authors also remove insignificant vocab from mBERT, keeping only Vietnamese vocab.

mBERT
mBERT followings the same training procedure of BERT (Devlin et al., 2018) on a multilingual dataset. The training languages are the 100 languages with the largest amount of text on Wikipedia, which includes Vietnamese 1 .

mBART
mBART (Liu et al., 2020) is a sequence-tosequence denoising encoder-decoder model that was pre-trained on large-scale monolingual corpora (including Vietnamese) using the BART objective and architecture .

mT5
mT5 (Xue et al., 2020) is a multilingual variant of the large encoder-decoder pre-trained text-to-text models T5 (Raffel et al., 2019). mT5 inherits all of the benefits of T5 including the scale and the design which was based on a large-scale empirical study (Raffel et al., 2019).

Methods
Following the works descried in (Vaswani et al., 2017), a transformer-based sequence-to-sequence architecture is an encoder-decoder architecture that has multi-layer self-attention. This architecture consists of two components: an encoder and a decoder.
Therefore, in order to generate a target sequence from an input sequence, a full transformer architecture with both an encoder and a decoder is required. PhoBERT and ViBERT are encoder-only models developed for encoding Vietnamese Language representations, so we can quickly assemble these models into the architecture. However, for the decoder level, we change the self-attention layers from bidirectional into left-context-only. We also insert a randomly initialized cross-attention mechanism into the decoder. We denote these models as PhoBERT2PhoBERT and ViBERT2ViBERT. Following the practice from (Press and Wolf, 2017), we tie the input embedding and output embedding in the decoder block, allowing the model to learn from the input embedding weights instead of random weights.
For existing encoder-decoder models like mT5 and mBART, we continuously fine-tune on the released checkpoints while strictly keeping their structure and vocabulary.

Datasets
We test the VieSum models on two datasets: Wikilingua (Ladhak et al., 2020) and Vietnews (Nguyen et al., 2019). The statistics for these datasets are shown in Table 1.

Wikilingua
Wikilingua (Ladhak et al., 2020) is a large-scale corpus multilingual corpus for abstractive summarization tasks. The corpus consists of 18 languages, including Vietnamese. These articles and summary pairs are extracted from WikiHow 2 . Wikihow is an online resource containing guidelines and how-to articles for a wide-range of topics. These articles are written by human authors; to ensure quality and accuracy, human experts reviewed the content before inclusion in this study 3 . Most of the non-English articles on the site are manually translated from the original English articles. Prior to publication, these foreign language articles are then reviewed by the WikiHow's international translation team.

Vietnews
Vietnews (Nguyen et al., 2019) collects news data from 3 well-known Vietnamese news websites: tuoitre.vn, vnexpress.net, and nguoiduatin.vn. The collected articles were written from 2016 to 2019. Then the authors remove all articles related to questionnaires, analytical comments and weather forecasts as they are not relevant to document summarization. The final corpus only contains news events.
6 Experiments Setup

Baselines
We verify the effectiveness of our proposed methods by comparing with the Transformer model architecture based on (Vaswani et al., 2017). The Transformer models has an encoder and a decoder, in which each layer of encoder-dencoder includes two major components: a multi-head self-attention and a feed-forward network. These model are intialized with random weights and labels RND.

ROUGE (Recall-Oriented Understudy for Gisting
Evaluation) measures the number of overlapping units (including n-grams and word sequences) between the generated summary and the reference summary.
• ROUGE-N: measures overlap between unigrams, bigrams, trigrams and higher order n-grams For our experiments, we will use ROUGE-1 and ROUGE-2 (unigrams and bigrams) • ROUGE-L: measures longest matching sequence of words using Longest Common Subsequences (LCS) with the assumption that longer LCS between the generated summary and the reference summary shows more similarity (higher quality of results).

Results
We report the results of transformer encoderdecoder models on two datasets: Wikilingua and Vietnews in Table 2.

Wikilingua
For the Wikilingua dataset, a first obvious take away is that a pre-trained decoder is important for a transformer model to perform well in Vietnamese  Summarization tasks. PhoBERT2RND has minor improvement comparing to untrained Transformer baseline model. Yet, incorporating a pre-trained decoder in PhoBERT2PhoBERT improves the understanding of a transformer model on a Vietnamese language and Summarization. The score improved from 2-4% accross for all metrics (Rouge-1, Rouge-2, and Rouge-L). Therefore, this setting can be further studied and improved by incorporating a larger pre-trained decoder like GPT (Brown et al., 2020) on Vietnamese language with existing pre-trained encoders. mT5 has the highest score across all metrics (Rouge-1, Rouge-2, and Rouge-L), following by mBART model. While PhoBERT2PhoBERT, which was trained monolingual on Vietnamese, has the lowest score compared to other pre-trained encoder-decoder models. This result can be attributed to a training data factor. Both mT5 and mBART train on mC4 (Raffel et al., 2019) and CC25  respectively, which are a extracted from a large Common Crawl corpus ( (Schwenk et al., 2020)) 4 . Common Crawl is a publicly web archive that provides text scraped from websites. These datasets (C4, CC25, and Common Crawl) may include Vietnamese Wikipedia languages website that help the model excel in constrained domain corpus (Wikipedia). We will 4 https://commoncrawl.org/ further discuss the affect of pre-training data for constrained domain corpus in Section 7.2.

Vietnews
Following the discussion of constrain domain training data in Section 7.1, PhoBERT2PhoBERT, which was trained on 20GB of news text, excels on the Vietnews corpus. On the other hand, large multingual model mT5 and mBART don't have a significant increase in scores compared to RND2RND baseline.
The importance of a pre-trained decoder also shows in the Vietnews corpus. PhoBERT2RND shows approximately the same result as the Transformer baseline.

Conclusion
In this manuscript, we test a series of availbale pre-trained Transformer on Vietnamese Abstractive Summarization Tasks. Following the transformer architecture proposed by (Vaswani et al., 2017), we showed that incorporating a pre-trained decoder to existed pre-trained encoder will improve the understanding of the model on the Vietnamese language and Summarization ability. We also show that pretraining data quality will affect the performance of the model on a constrain domain.