{'fraction_non_alphanumeric': 0.03873039863971283, 'fraction_numerical': 0.019270734932930286, 'mean_word_length': 4.546102263202012, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Are the Multilingual Models Better? Improving Czech Sentiment with Transformers

In this paper, we aim at improving Czech sentiment with transformer-based models and their multilingual versions. More concretely, we study the task of polarity detection for the Czech language on three sentiment polarity datasets. We fine-tune and perform experiments with five multilingual and three monolingual models. We compare the monolingual and multilingual models' performance, including comparison with the older approach based on recurrent neural networks. Furthermore, we test the multilingual models and their ability to transfer knowledge from English to Czech (and vice versa) with zero-shot cross-lingual classification. Our experiments show that the huge multilingual models can overcome the performance of the monolingual models. They are also able to detect polarity in another language without any training data, with performance not worse than 4.4 % compared to state-of-the-art monolingual trained models. Moreover, we achieved new state-of-the-art results on all three datasets.


Introduction
In recent years, BERT-like models (Devlin et al., 2019) based on the Transformer architecture (Vaswani et al., 2017) and generalized language models brought a significant improvement in performance in almost any NLP task (Raffel et al., 2020a), especially in English. Despite this fact, not much work has been recently done in sentiment analysis for the Czech language with the latest Transformer models. We partly fill this gap by focusing on the Sentiment Classification task, also known as Polarity Detection.
Polarity detection is a classification task where the goal is to assign a sentiment polarity to a given text. The positive, negative and neutral classes are usually used as the polarity labels. The polarity can also be defined with a different number of labels, i.e., fine-grained sentiment analysis (Liu, 2012).
Our motivation is to reveal the performance limits of the current SotA transformer-based models on the Czech polarity detection task, check the ability of the multilingual models to transfer knowledge between languages and unify the procedure and data that enable the correct future evaluation of this task.
In this paper, we focus on the task of polarity detection applied on Czech text by comparing the performance of seven pre-trained transformer-based models (both monolingual and multilingual) on three Czech datasets. We fine-tune each model on each dataset and we provide a comprehensive survey of their performance. Our experiments show the effectiveness of the Transformer models that significantly outperform the older approaches based on recurrent neural networks. We observe that the monolingual models can be notably outperformed by the multilingual models, but only by those with much more parameters. Moreover, we achieve new state-of-the-art results on all three evaluated datasets.
We are also interested in the ability of the multilingual models to transfer knowledge between languages and its usability for polarity detection. Thus, we perform zero-shot cross-lingual classification, fine-tune four cross-lingual transformer-based models on the English dataset and then test the models on Czech data. We also perform the same experiment in the reverse direction, i.e., from Czech to English. The results reveal that the XLM-R-Large model (fine-tuned solely on English) can achieve very competitive results that are only about 4 % worse than the SotA model fine-tuned by us on Czech data. To the best of our knowledge, this is the first paper that performs zero-shot cross-lingual polarity detection for the Czech language.
We also noticed that the comparison with the previous works is rather problematic and thus, we provide a split for all Czech datasets that allows comparing future works much easier. Our code and pre-trained models are publicly available 1 .
Our main contributions are the following: 1) We provide the comprehensive performance comparison of the currently available transformer-based models for the Czech language on the polarity detection task along with the models' optimal settings. 2) We test the ability of the multilingual models to transfer knowledge between Czech and English.
3) We release all the fine-tuned models and code freely for research purposes and we provide a data split that allows future comparison and evaluation. Furthermore, we achieved new state-of-the-art results for all three evaluated datasets.

Related Work
The previous approaches (Kim, 2014;Johnson and Zhang, 2016;Cliche, 2017;Baziotis et al., 2017;Gray et al., 2017;Conneau et al., 2017) for English polarity detection and other related tasks mostly relied on transfer learning and pre-trained word embeddings such as word2vec (Mikolov et al., 2013) and fastText (Bojanowski et al., 2017) in combinations with Convolutional Neural Networks (CNN) or Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), eventually in conjunction with the modified attention mechanism (Bahdanau et al., 2015;Rocktäschel et al., 2015;Raffel and Ellis, 2015). Furthermore, the new contextualized word representations such as CoVe (Mc-Cann et al., 2017) or ELMo (Peters et al., 2018) and pre-trained language model ULMFiT (Howard and Ruder, 2018) were successfully applied to the polarity detection. Finally, the latest transformerbased models like BERT (Devlin et al., 2019), GPT (Radford et al., 2018), RoBERTa  or T5 (Raffel et al., 2020b) that are all in general trained on language modeling tasks proved their performance superiority for English over all previous approaches, for example in (Sun et al., 2019). These models are pre-trained on a modified language modeling tasks with a huge amount of unlabeled data. In the end, they are fine-tuned for a specific downstream task.
The initial works on Czech polarity detection and sentiment analysis usually used lexical features (Steinberger et al., 2011;Veselovská et al., 2012) or Bag-of-Words text representations along with the Naive Bayes or logistic regression classifiers (Habernal et al., 2013) or a combination of supervised and unsupervised approach (Brychcín and Habernal, 2013). Lenc and Hercig (2016) applied CNN using the architecture from (Kim, 2014) and the LSTM neural network to all three datasets that we use in this paper. Another usage of LSTM neural network with the self-attention mechanism (Humphreys and Sui, 2016) can be found in (Libovickỳ et al., 2018). Similarly, Sido and Konopík (2019) tried to use curriculum learning with CNN and LSTM. Lehečka et al. (2020) pre-trained a BERT-based model for polarity detection with an improved pooling layer and distillation of knowledge technique. The most recent application of the Transformer model is in (Sido et al., 2021). The authors created a BERT model for Czech and, as one of the evaluation tasks, they performed polarity detection on the FB and CSFD datasets.
To the best of our knowledge, there are no previous works that focus on the zero-shot cross-lingual polarity detection task in the Czech language. The recent related work can be found in (Eriguchi et al., 2018), where the authors use the neural machine translation encoder-based model and English data to perform zero-shot cross-lingual sentiment classification on French. In (Eriguchi et al., 2018) the authors performed the zero-shot classification from Slovene to Croatian. Another related work can be found in (Wang and Banko, 2021;Qin et al., 2020).

Data
To the best of our knowledge, there are three Czech publicly available datasets for the polarity detection task: (1) movie review dataset (CSFD), (2) Facebook dataset (FB) and (3) product review dataset (Mallcz), all of them come from (Habernal et al., 2013) and each text sample is annotated with one of three 2 labels, i.e., positive, neutral and negative, see Table 1 for the class distribution. For the crosslingual experiments we use the two-class English movie review dataset (IMDB) (Maas et al., 2011  The FB dataset contains 10k random posts from nine different Facebook pages that were manually annotated by two annotators. The CSFD dataset is created from 90k Czech movie reviews from the Czech movie database 3 that were downloaded and annotated according to their star rating (0-2 stars as negative, 3-4 stars as neutral, 5-6 stars as positive). The Mallcz dataset consists of 145k users' reviews of products from Czech e-shop 4 , the labels are assigned according to the review star rating on the scale 0-5, where the reviews with 0-3 stars are labeled as negative, 4 stars as neutral and 5 stars as positive. The English IMDB dataset includes 50k movie reviews scraped from the Internet Movie Database 5 with positive and negative classes split into training and testing parts of equal size.
Since there is no official partitioning for the Czech datasets, we split them into training, development and testing parts with the same class distribution for each part as it is in the original dataset, see Table 1. For the Mallcz and CSFD datasets, we use the following ratio: 80 % for training data, 20 % for testing data, for the FB dataset, it is 70 % and 30 %, respectively and 10 % from the training data (for all datasets) is used as the development data. We used different split ratio for the FB dataset because it is approximately ten and sixteen times smaller than the CSFD and Mallcz datasets, respectively and we did not want to reduce the size of the testing data too much.

Models Description
We performed exhaustive experiments with transformed-based models and in order to compare them with the previous works, we also implemented the older models (baseline models) that include the logistic regression classifier and the BiLSTM neural network.

Baseline Models
We re-implemented the best models from (Habernal et al., 2013), i.e., logistic regression classifier (lrc) with character n-grams (in a range from 3grams to 6-grams), word uni-grams and bi-grams features. The second baseline model is the LSTM model partly inspired by (Baziotis et al., 2017). Its input is a sequence of t tokens represented as a matrix M ∈ R t×d , where d = 300 is a dimension of the Czech pre-trained fastText word embeddings (Bojanowski et al., 2017) 6 . The maximal size of the input vocabulary is set to 300 000. The input is passed into the trainable embedding layer that is followed by two BiLSTM (Graves and Schmidhuber, 2005) layers and after each, the dropout (Srivastava et al., 2014) is applied. After the two BiLSTM layers, the self-attention mechanism is applied. The output is then passed to a fully-connected softmax layer. An output of the softmax layer is a probability distribution over the possible classes. We use the Adam (Kingma and Ba, 2014) optimizer with default parameters (β 1 = 0.9, β 2 = 0.999) and with weight decay modification (Loshchilov and Hutter, 2017) and the cross-entropy loss function. We replace numbers, emails and links with generic tokens, we tokenize input text with the TokTok tokenizer 7 and we use a customized stemmer 8 .
We use different hyper-parameters for each dataset, see Appendix A.1 for the complete settings.

Transformer Models
In total, we use eight different transformer-based models (five of them are multilingual). All of them are based on the original BERT model. They use only the encoder part of the original Transformer (Vaswani et al., 2017), although their pre-training procedure may differ. There are also text-to-text models like T5 (Raffel et al., 2020b) and BART  and their multilingual versions mT5 (Xue et al., 2020) and mBART (Liu et al., 2020;Tang et al., 2020). The main difference from BERT-like models is that they use the full encoderdecoder architecture of the Transformer. They are mainly intended for text generation tasks (e.g., abstractive summarization). We decided to use only the BERT-like models with the same architecture because they fit more for the classification task.
All the models are pre-trained on a modified language modeling task, for example, Masked Language Modeling (MLM) and eventually on some classification task like Next Sentence Prediction (NSP) or Sentence Ordering Prediction (SOP), see (Devlin et al., 2019;Lan et al., 2020) for details. The evaluated models differ in the number of parameters (see Table 2) and thus, their performance is also very different, see Section 5.

Model
#Params Vocab #Langs Czert-B is Czech version of the of the original BERT BASE model (Devlin et al., 2019). The only difference is that during the pre-training, the authors increased the batch size to 2048 and they slightly modified the NSP prediction task (Sido et al., 2021).
Czert-A is the Czech version of the ALBERT model (Lan et al., 2020), also with the same modification as Czert-B, i.e., batch size was set to 2048 and the modified NSP prediction task is used instead of the SOP task (Sido et al., 2021).
RandomALBERT we follow the evaluation in (Sido et al., 2021) and we also test randomly initialized ALBERT model without any pre-training to show the importance of pre-training of such models and its performance influence on the polarity detection task.

Transformers Fine-Tuning
To utilize the models for text classification, we follow the default approaches mentioned in the corresponding models' papers and we fine-tune all parameters of the models. In all models except XLM, we use the final hidden vector h ∈ R H of the special classification token [CLS] or <s> taken from the pooling layer 9 of BERT or RoBERTa models, respectively. The vector h represents the entire encoded sequence input, where H is the hidden size of the corresponding model. We add a task-specific linear layer (with a dropout set to 0.1) represented by a matrix W ∈ R |C|×H , where C is a set of classes. We compute the classification output, i.e., the input sample being classified as class c ∈ C as c = argmax (hW T ).
In the case of the XLM model, we take the last hidden state (without any pooling layer) of the first input token and we apply the same linear layer (W ∈ R |C|×H ) and approach to obtain the classification output. For learning, we use the Adam optimizer with default parameters and with weight decay (same as for the LSTM model), and the cross-entropy loss function. See Section 5.1 and Appendix A.2 for the hyper-parameters we used.

Experiments & Results
We perform two types of experiments, i.e., monolingual and cross-lingual. In monolingual experiments, we fine-tune and evaluate the Transformer models for each dataset separately on three-class (positive, negative and neutral) and two-class (positive and negative) sentiment analysis. We also implemented the logistic regression (lrc) and LSTM baseline models and we compare the results with the existing works.
In cross-lingual experiments, we test the ability of four multilingual transformer-based models to transfer knowledge between English and Czech. We run the multilingual models only on the twoclass datasets (positive and negative). We fine-tune either on English (IMDB) or Czech (CSFD), and then we evaluate on the other language. Thus we perform the zero-shot cross-lingual classification. We decided to use the IMDB and CSFD dataset because they are from the same domain i.e., movie reviews.
Each experiment 10 was performed at least five times and we report the results using the macro F 1 score.

Monolingual Experiments
The goal of the monolingual experiments is to reveal the current state-of-the-art performance on the Czech polarity datasets, namely CSFD, FB and Mallcz (see Section 3) and provide a comparison between the available models and their settings.
As we already mentioned, we split the datasets into training, development and testing parts. There is no official split for the datasets and we found out that all the available works usually use either 10-fold cross-validation or they split 11 the datasets on their own, the † and * symbols in Table 3, respectively causing the comparison to be difficult.
We fine-tune all models on training data and we measure the results on the development data. We select the model with the best performance on the development data and we fine-tune the model on combined training and development data. We report the results in Table 3 on the testing data with 95% confidence intervals. 10 Except for the experiments with the lrc model. 11 The authors do not provide any recipe to reproduce the results.
Firstly, we re-implemented the logistic regression classifier (lrc) with the best feature combination from (Habernal et al., 2013) and we report the results on our data split. We can see that we obtained very similar results to the ones stated in (Habernal et al., 2013). We also tried to improve this baseline with Tf-idf weighting, but it did not lead to any significant improvements, so we decided to keep the settings the same as in (Habernal et al., 2013), so the results are comparable.
For the LSTM model, we tried different combinations of hyper-parameters (learning rate, optimizer, dropout, etc.). We report the used hyperparameters for the results from Table 3 in Appendix A.2. Our implementation is only about 1 % worse than LSTM with the self-attention model from (Libovickỳ et al., 2018), but they used a different data split. For the Mallcz dataset, we were not able to outperform the lrc baseline with the LSTM model.
We fine-tune all parameters of the seven pretrained BERT-based models and one randomly initialized ALBERT model. In our experiments, we use constant learning rate and also linear learning rate decay (without learning rate warm-up) with the following initial learning rates: 2e-6, 2e-5 and 2.5e-5. We got inspired by the ones used in (Sun et al., 2019). Based on the average number of tokens for each dataset and models' tokenizer (see Table 4 and Figures 1, 2, 3) 12 , we use a max sequence length of 64 and a batch size of 32 for the FB dataset. We restrict the max sequence length for the CSFD and Mallcz datasets to 512 and use a batch size of 32. All other hyper-parameters of the models are set to the pre-trained models' defaults. See Table 7 in Appendix A.2 for the reported results' hyper-parameters.
We repeated all the basic experiments with the polarity detection task from (Sido et al., 2021) with the new data split. Our results do not significantly differ, as shown in Table 8 and in Appendix A.2. If we compare the BERT model from (Lehečka et al., 2020) with the Czert-B, mBERT and SlavicBERT models 13 , we can see that on the binary task, they also perform very similarly, i.e., around 93 %, but again they used different test data (the entire CSFD dataset 14 ). The obvious observation is that the XLM-R-Large model is supe- 12 The distributions of the other models were similar to those shown in the mentioned Figures. 13 All of them should have the same or almost the same architecture and a similar number of parameters.
14 The examples with positive and negative classes.   rior to all others by a significant margin for any dataset. Only for the three-class Mallcz dataset, the Czert-B model is competitive (the confidence intervals almost overlap). From the results for the RandomALBERT model, we can see how important is the pre-training phase for Transformers, since the model is even worse than the logistic regression classifier 15 .

Cross-lingual Experiments
The cross-lingual experiments were performed with the multilingual models that support English and Czech. For these experiments, we use linear learning rate decay with an initial learning rate of 2e-6.
Firstly, we fine-tuned the models on the English IMDB dataset and we evaluated them on the test part of the Czech binary CSFD dataset (i.e., zero-15 The model was trained for a maximum of 15 epochs and it would probably get better with a higher number of epochs, but the other models were trained for the same or lower number of epochs. shot cross-lingual classification). We randomly selected 5k examples from the IMDB dataset as the development data. The rest of the 45k examples is used as training data. We select the models that perform best on the English development data 16 and we report the results in Table 5. The test (cs) column refers to results obtained on the CSFD testing part. For easier comparison, we also include the Monoling. (cs) column that contains the results (same as in Table 3) for models trained on Czech data. The XLM-R-Large was able to achieve results only about 4.4 % worse than the same model that was fine-tuned on Czech data. It is a great result if we consider that the model has never seen any labeled Czech data. The XLM and mBERT models perform much worse.  The second type of cross-lingual experiment was performed in a reverse direction, i.e., from Czech to English. We use the Czech CSFD training and testing data for fine-tuning and we evaluate the model on the English IMDB test data. We report the results in Table 6 using the accuracy because the current state-of-the-art works (Thongtan and Phienthrakul, 2019;Sun et al., 2019) use this metric. Similarly to the previous case, we selected the model that performs best on Czech CSFD development data. For these experiments, the mBERT did not converge. As in the previous experiment, the XLM-R-Large performs best and it achieves almost 94 % accuracy that is only 3.4 % below the current SotA result from (Thongtan and Phienthrakul, 2019  Based on the results, we can conclude that the XLM-R-Large model is very capable of transferring knowledge between English and Czech (and probably between other languages as well). It is also important to note that Czech and English are languages from a different language family with a high number of differences both in syntax and grammar.

Discussion & Remarks
We can see from the results that the recent pretrained transformer-based models beat the older approaches (lrc and LSTM) by a large margin. The monolingual Czert-B model is in general outperformed only by the XLM-R-Large and XLM-R-Base models, but these models have five times/three times more parameters, and eight times larger vocabulary. Taking into account these facts, the Czert-B model is still very competitive. It may be beneficial in certain situations to use a smaller model like this that does not need such computational resources as the ones that are required by the XLM-R-Large.
During the fine-tuning, we observed that in most cases, the lower learning rate 2e-6 (see Table 7 in Appendix A.2) leads to better results. Thus we recommend using the same one or similar order. The higher learning rates tend to provide worse results and the model does not converge.
According to the generally higher confidence interval, the fine-tuning of a smaller dataset like FB that has only about 6k training examples is, in general, less stable and more prone to overfitting than training a model on datasets with tens of thousands of examples. We also noticed that fine-tuning of the mBERT and SlavicBERT on the Mallcz dataset is very unstable (see the confidence interval in Table 3). Unfortunately, we did not find out the reason. A more detailed error analysis could reveal the reason.

Conclusion
In this work, we evaluated the performance of available transformer-based models for the Czech language on the task of polarity detection. We compared the performance of the monolingual and multilingual models and we showed that the large XLM-R-Large model can outperform the monolingual Czert-B model. The older approach based on recurrent neural networks is surpassed by a very large margin by the Transformers. Moreover, we achieved new state-of-the-art results on all three Czech polarity detection datasets.
We performed zero-shot cross-lingual polarity detection from English to Czech (and vice versa) with four multilingual models. We showed that the XLM-R-Large is able to detect polarity in another language without any labeled data. The model performs no worse than 4.4 % in comparison to our new state-of-the-art monolingual model. To the best of our knowledge, this is the first work that aims at cross-lingual polarity detection in Czech. Our code and pre-trained models are publicly available.
In the future work, we intend to perform a deep error analysis to find in which cases the current models fail and compare approaches that use the linear cross-lingual transformations (Artetxe et al., 2018;Brychcín, 2020) that explicitly map semantic spaces into one shared space. The second step in the cross-lingual settings is to employ more than two languages and utilize the models for different domains. For all datasets, we use 10 % of total steps (batch updates) to warm up the learning rate, which means that during the training, the linear rate is firstly linearly increasing to the initial learning rate before being decayed with the corresponding learning rate scheduler. The dropout after the BiLSTM layers is set to 0.2. We use cosine (the * symbol in Table  7) and the exponential learning rate scheduler (the ‡ symbol in Table 7) with a decay rate set to 0.05. Table 7 contains the initial learning rate and the number of epochs for the LSTM model for each dataset.

A.2 Transformer Hyper-parameters
For fine-tuning of the transformer-based models, we use the same modification (Loshchilov and Hutter, 2017) of the Adam (Kingma and Ba, 2014) optimizer with default weight decay set to 1e-2.
We use different learning rates and a number of epochs for each combination of the models and datasets, see Table 7. We use either constant linear rate or linear learning rate decay without learning rate warm-up. We use default values of all other hyper-parameters. For the cross-lingual experiments, we use only the linear learning rate decay scheduler with the initial learning rate set to 2e-6 without learning rate warm-up. For the cross-lingual experiments from English to Czech, the numbers of epochs used for the fine-tuning are 5, 2, 4 and 10 for the XLM-R-Base, XLM-R-Large, XLM and mBERT models, respectively. For the cross-lingual 17 Available at https://fasttext.cc/docs/en/ crawl-vectors.html experiments from Czech to English, the numbers of epochs used for the fine-tuning are 25, 5 and 9 for the XLM-R-Base, XLM-R-Large and XLM models 18 , respectively.

A.3 Computational Cluster
For fine-tuning of the Transformers models we use the Czech national cluster Metacentrum 19 . We use two NVIDIA A100 GPUs each with 40GB memory.   (Sido et al., 2021) with results obtained by us.