{'fraction_non_alphanumeric': 0.028149163905077514, 'fraction_numerical': 0.008113358473317334, 'mean_word_length': 4.929087624209576, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A large-scale computational study of content preservation measures for text style transfer and paraphrase generation

Text style transfer and paraphrasing of texts are actively growing areas of NLP, dozens of methods for solving these tasks have been recently introduced. In both tasks, the system is supposed to generate a text which should be semantically similar to the input text. Therefore, these tasks are dependent on methods of measuring textual semantic similarity. However, it is still unclear which measures are the best to automatically evaluate content preservation between original and generated text. According to our observations, many researchers still use BLEU-like measures, while there exist more advanced measures including neural-based that significantly outperform classic approaches. The current problem is the lack of a thorough evaluation of the available measures. We close this gap by conducting a large-scale computational study by comparing 57 measures based on different principles on 19 annotated datasets. We show that measures based on cross-encoder models outperform alternative approaches in almost all cases.We also introduce the Mutual Implication Score (MIS), a measure that uses the idea of paraphrasing as a bidirectional entailment and outperforms all other measures on the paraphrase detection task and performs on par with the best measures in the text style transfer task.


Introduction
Text style transfer (TST) and paraphrases generation (PG) are active areas of research in NLP, with dozens of papers proposing new methods. These methods could be applied for practical purposes, such as supporting human writers, personalizing digital assistants, or even creating artificial personalities.
Research and development of TST models require fast feedback loops, and they require fast and reliable automatic quality measures. TST is hard to evaluate for several reasons. First, golden answers, even if available, are not the only valid way to rewrite the text. Second, parallel corpora with different styles do not emerge naturally and are hard to find. This means that reference-based evaluation is often prohibitive and creates a need for manual evaluation of TST or for clever automatic measures.
The basic desired properties of TST are style accuracy, content preservation, and fluency (Mir et al., 2019). For many methods of unsupervised TST, keeping the content of the original text and automatically measuring its preservation is one of the most difficult tasks (see e.g. Dale et al. (2021)).
During development, the only way to control content preservation is to use automatic measures. Such measure takes two sentences and return the value which indicates the similarity of their content. More formally, the measure sim quantifies semantic relatedness of two utterances, an original text x and a style-transferred or paraphrased text y : sim(x, y)→[0; 1]. The measure sim yields high score for the pairs with similar content and low score for ones with different content.
As Krishna et al. (2020) and Yamshchikov et al. (2021) show, most TST works evaluate the content preservation with BLEU (Papineni et al., 2002) or similar measures based on word overlap between two texts. The situation in PG is almost identical. Most works including the most recent ones (Sun et al., 2021;Fu et al., 2020) also still rely on BLEU.
Even though measures like BLEU, based on a word or character-level n-grams are pretty intuitive and straightforward, they don't take into account synonyms and distributively related words. Moreover, there already exist several pieces of evidence that correlation of standard BLEU-like automatic measures is relatively low (Briakou et al., 2021). The recent development of vector representations of textual information (Mikolov et al., 2013;Zhang et al., 2019) and various ways to handle these vectors provides room for improvement of the approaches to scoring the content preservation. It is, therefore, crucial to perform a thorough analysis of all existing content preservation measures and to gather best practices from the top-performing approaches to create a new approach that could demonstrate stable performance in terms of both PG and TST tasks.
In this work, we further extend a comprehensive study of Yamshchikov et al. (2021) by analyzing a much more diverse set of measures including recently developed transformers-based ones, and also by proposing a new measure specially developed for TST and PG content preservation scoring. The contributions of our paper are as follows: • We perform a large-scale evaluation of automatic content preservation measures for text style transfer and paraphrase generation tasks, which includes 57 measures applied to 9 paraphrasing datasets and 10 text style transfer datasets. To the best of our knowledge, this is the largest and the most comprehensive evaluation of this kind; • We introduce Mutual Implication Score (MIS): a measure of content preservation based on predictions of NLI models in two directions. We show that it outperforms all known measures in paraphrase detection and shows consistently high results for TST. We opensource the model on Huggingface Model Hub. 1 The code for measures and experiments is released publicly. 2 2 Related work

Measures of content preservation
There exists a large number of content preservation measures that can be classified into several groups. In this section, we describe all of these approaches. Refer to Figure 1 for a schematic description of all approaches.
Words or characters n-grams (ngram) The most simple and intuitive way to compare two texts is based on the overlap of word or character n-grams. The standard method used to evaluate the quality of a generated text is to compare it with a human-written reference text via BLEU 1 https://huggingface.co/ SkolkovoInstitute/Mutual_Implication_ Score 2 https://github.com/skoltech-nlp/ mutual_implication_score score (Papineni et al., 2002), which is the precision of word n-grams. In TST and PG papers, BLEU is often used to evaluate content preservation relative to the original text or a reference. Other popular measures based on words or n-grams are ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), chrF (Popović, 2015). Such approaches as Levenshtein distance (Levenshtein et al., 1966), Jaro-Winkler distance (Jaro, 1989) also work at the subword level by calculating the edit distance between two sequences, so we also refer them to the ngram group. Panchenko and Morozova (2012) provided a comparative study of classic word similarity measures and their combinations. The ngram measures are simple and intuitive but do not handle well such linguistic phenomena as synonyms, negation, and issues with word order.
Similarity between static embeddings (embstatic) Another family of measures partially overcomes these difficulties by representing texts with their embeddings and calculating the distance (e.g. cosine similarity) between the embeddings of two texts. This group of measures can be further divided by the way the embeddings are generated. The basic way of obtaining the embedding of a text is by averaging across static word embeddings: Word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), FastText (Bojanowski et al., 2017).
Similarity between embeddings from biencoders (emb-bi-enc) Embeddings of a text can be generated by encoding a text with a pre-trained encoder. If the two texts are encoded separately, and then we compute the cosine similarity between their embeddings, we refer to such models as bi-encoders. This group of models is usually trained in a supervised manner. The encoders can be trained on the translation task (Laser (Artetxe and Schwenk, 2019), LaBSE (Feng et al., 2020)), paraphrase identification task (SIMILE (Wieting et al., 2019)), or text generation task (BARTScore (Yuan et al., 2021)). They potentially can compare the meanings of texts that are very different in terms of structure and vocabulary.  Symmetric and asymmetric cross-encoders (sym/asym-cross-enc) The models called crossencoders process both texts simultaneously using cross-attention and directly predict the relationship between the texts. They can perform symmetrically (score is independent of the order of the texts being compared) or asymmetrically (score strongly depends on the order of the texts). Due to their supervised nature, such models can reflect content preservation more accurately than word-based approaches, but they depend on labeled data and may not generalize well to new domains. The presence of symmetry is defined by the task the model was trained on. Thus, models trained on the Natural Language Inference (NLI) task data (such as BLEURT (Sellam et al., 2020)  suggest a similar model for evaluation of summarization and text style transfer. The main idea of these works is to use NLI models in a two-fold manner (direct and reverse). NLI models are generally asymmetric cross-encoders, so we classify this group of approaches as a two-fold asymmetric encoder. As shown in Figure 2, despite the wide variety of measures, n-gram-based measures are still used most often, while embedding-based measures and cross-encoders are much less popular. In some papers, no automatic content preservation measures are used.

Evaluation of content preservation measures
Our work in many respects follows the setup of Yamshchikov et al. (2021) (Table 7). measures. They evaluated the quality of automatic measures by the correlation between rankings provided by these measures and rankings created by human scores. This scoring showed that the WMD over GloVe embeddings and L2 distance between the ELMo embeddings outperform other measures. However, no supervised sentence encoders or crossencoders were considered in this work.
In the work by Briakou et al. (2021), the authors evaluated measures of formality transfer in four languages. The main subject of this work is a thorough analysis of multilingual formality style transfer, including a high-level analysis of all aspects of style transfer quality: style accuracy, content preservation, and fluency. The authors used chrF and a cross-encoder (XLM-R) trained on a semantic text similarity dataset to calculate content preservation. They also cautioned against using BLEU in this context, because it has a lower correlation with human judgments than many other measures. However, automatic measures of content preservation were not the main focus of this work, so we extend its results by applying more diverse measures on the English part of their dataset, among others.

Datasets used in comparative study
We run our analysis of measures on parallel datasets manually labeled for semantic similarity or content preservation. To make the comparison more generalizable, we fetch a large number of datasets generated by different models.

Text style transfer datasets
The text style transfer task is aimed at transforming a text to change its style (a particular attribute of its text) while keeping the content intact. Since in some cases the style cannot be separated from the content (e.g. if the style is positive/negative sentiment), strict preservation of all content is sometimes impossible in the TST task. Therefore, we consider the parallel TST datasets separately from other data used for the analysis.
In many TST works, outputs were evaluated with human judgments, but the raw similarity labels are rarely published. We managed to find datasets that include human similarity scores for various TST tasks

Paraphrases datasets
Unlike TST, the paraphrase generation task requires full preservation of content. There exist a large number of parallel datasets of paraphrases manually labelled for content preservation. The majority of them have binary labels ("same"/"different"). We use the following datasets in our analysis:  • APT (Nighojkar and Licato, 2021), • Yam. Para (Yamshchikov et al., 2021).
We provide detailed information about the datasets in the Appendix tables 5 and 6.

Mutual Implication Score (MIS)
The goal of our research is not only to analyze the existing measures of content preservation but also to suggest a new measure that can outperform the existing ones. We devise a new measure that is based on measuring content similarity with NLI, as described by Nighojkar and Licato (2021). In this work, the authors exploit the assumption that implies the two sentences with the same meaning should be equivalent in their inferential properties, i.e. each sentence should textually entail the other. This means that the NLI model is supposed to return similar entailment scores when applied to semantically equal sentences regardless of the sequence these sentences are sent to the input of the model. The authors used this assumption to propose an adversarial method of dataset creation for paraphrase identification.
NLI models predict whether one text logically entails another, and are, therefore, asymmetric. High entailment probability in the forward direction means that the second text accurately follows the first one and does not contain hallucinated information. A high entailment score in the backward direction means that all the information from the first text is retained in the second text.
The most natural way to aggregate scores from both directions is to multiply them or compute their arithmetic or harmonic mean. We use this approach as a baseline. We yield NLI scores from the following models:  Although these NLI models are a good starting point, they might not be fully suitable for measuring content preservation, because they were trained for a different task. We suggest that further finetuning them on the data annotated with content preservation scores might yield better models.
Thus, we modify the RoBERTA architecture used for NLI. Namely, we use the original encoder in both forward and backward directions, concatenate the last hidden states, and then send them to the classification module which is tuned on data annotated with content preservation scores. We refer to this model as Mutual Implication Score (MIS). The scheme of our model is given in Figure 3. We initialize the model with pre-trained weights from the RobNLI model. We tune it on Quora Question Pairs dataset (Sharma et al., 2019) for 2 epochs with a learning rate 4e −6 and all but the last encoder layer and classifier layer frozen.
We evaluate the model with the Spearman rank correlation coefficient of the automatic content preservation scores with human judgments. We evaluate all TST and PG datasets introduced in Section 3. We evaluate MIS and baseline NLIbased measures (we aggregate the NLI scores for both directions with the arithmetic mean because it showed the best results in our preliminary experiments).
The results are shown in Table 1. Fine-tuning the (slightly modified) NLI model on content preser-vation data slightly improves its performance on datasets generated by paraphrasing models and yields significantly higher correlation on TST datasets.

Measures analysis
We compute the content preservation scores for paraphrasing and style transfer datasets using measures of different types. We analyze the performance of individual measures and compare the performance of different groups of measures. We also look into the difference in measures performance on PG and TST tasks and analyze the individual datasets.

Experimental setting
We analyze 57 content preservation measures of different types. As described in Section 2.1, the measures can be divided into the following groups: a word or character n-gram based (ngram), the measures based on the distance between static (embstatic) or contextualized (emb-context) embeddings, or embeddings from bi-encoders (emb-bienc), different groups of encoders-based measures: symmetric (sym-cross-enc), asymmetric (asymcross-enc) or two-fold asymmetric (2x-asym-crossenc) cross-encoders. This grouping is used explicitly during analysis. The full list of measures is given in Table 8.
We compute the content preservation scores for 19 datasets listed in Section 3. The full information about the datasets is given in Appendix Tables 5  and 6.
We evaluate measures using the Spearman rank correlation coefficient of the automatic scores with human judgments. Since we use a large number of measures and datasets, we report only aggregated results. The full results are available in the Appendix Figures 7 and 8. Figure 4 shows the correlations of the bestperforming measures from different groups for individual datasets. The last columns of the plots show the performance of each measure averaged across datasets. The plot shows that MIS and similar measures based on two-folded asymmetric crossencoders have the best average performance on the paraphrase datasets. For TST datasets, there is no clear winner: symmetric cross-encoders (crossstsb-large/base), bi-encoders (SIMCSE-SL/SB),   Table 3: Spearman correlations of measures belonging to different groups: ρ max -correlation of the best-performing in the group, ρ avg -correlation averaged over the group, #wins -the number of times the model from the group performs best on any of the datasets.
The performance of content preservation measures on TST datasets varies from style to style. The TST datasets we use contain style transformations of four types: detoxification, formal to informal, positive to negative sentiment, and modern to old-style English. Thus, it seems natural to average the measures performance not only by all TST datasets but also by TST datasets of different styles. The averaged scores are shown in Table 2

Group-level analysis
To get more generalizable results of the analysis, we perform a group-level comparison of measures in Table 3. We report the Spearman correlation scores averaged over datasets of PG and TST tasks (as before, we do not merge all datasets and consider the two tasks separately). We report the mean and maximum correlations of all measures of a group. We also compute the number of times when a measure of a group performs best on the particular dataset. This indicator can be somewhat biased due to the nature of each dataset, however, it can still serve as an additional source of information.
If the difference between correlations is not significant (by Williams test (Graham and Baldwin, 2014)) we assign one winning time to each group. From this point of view, we can even better see that two-folded asymmetric models are the best choice for paraphrases detection because the mean correlation outperforms the next best-performing group by 0.05. Symmetric cross-encoders can also be an alternative option for this task because they show the largest number of wins. Symmetric crossencoders show the highest mean correlation on the TST task. At the same time, the number of wins and correlations of the best models from this class are similar for all encoder-based classes.
Finally, from the measure-level and group-level perspective, we can see that encoder-based measures outperform ngrams-based measures in the absolute majority of datasets on TST and PG tasks.

Data-level analysis
So far we relied on the correlations averaged across different datasets. However, it is also natural to have a closer look at how the behavior of different measures changes across datasets.
For this purpose, we represent each dataset as a vector of correlations of each measure with the human judgments and plot a dendrogram (see Figure 5) to show the clustered structure of the obtained vectors. The dendrogram should be interpreted as follows. The height at which each dataset is connected to another dataset or group of datasets indicates the distance between the dataset vectors. We additionally plot a heatmap of cosine similarities of these datasets vectors in Appendix Figure 9.
Datasets related to sentiment transfer (PG-YELP, Yam. Yelp) look different from others, thus, they form a separate cluster in the dendrogram. The reason for this dissimilarity is probably the fact that in this type of TST task (sentiment transfer) the content of the utterance changes more significantly than in other tasks. Moreover, PG-YELP is originally distributed as a pairwise comparison dataset. To yield sentence-level scores, we apply Luce Spectral Ranking (Maystre and Grossglauser, 2016). This preprocessing might affect the quality of labels.
In general, the datasets are clustered into two rather dense groups and this clustering does not match the separation of the datasets among TST and PG tasks. The different behavior of the tested measures might be explained by the way the data is annotated. For example, the PAWS datasets were collected in an adversarial manner (by shuffling the words in sentences), STRAP datasets were generated with TST models, and Yam. datasets were annotated by a similar group of workers -these three sets form clusters in the dendrogram.
6 Using automatic measures to rank text style transfer systems While above we compared automatic and human ranking of individual text pairs, our final goal is to find a measure to rank TST or PG systems. Six TST datasets used in our analysis were created by running several TST models on the same dataset and manually assessing the degree of content preservation in the resulting text pairs. They cover diverse tasks: formality transfer (xformal-FoST and STRAP_form datasets), text detoxification (Tox600 and CAE datasets), Shakespeare style transfer (STRAP_SP), and sentiment transfer (PG-YELP). We use the human judgments on content preservation from these datasets to rate the ability of various measures to rank text style transfer systems.
For brevity and clarity, we do not report the results of this analysis for all measures. Instead, we select the best-performing measure from each group: • cross-encoders: MIS, RobNLI/mean, BLEURT-L128 and cross-stsb-base, • bi-encoders: LaBSE and SIMCSE-SL (supervised, using ROBERTa-large),  Table 4: Mean rank correlation (ρ) of text style transfer system-level automatic scores with human judgments, and percentage of cases when they correctly identify the best system (acc).
We show the results aggregated across the datasets in Table 4. The scores for individual datasets and measures and a list of measures managed to identify the best-performing model for a given dataset are given in Appendix C.
No measure can fully match the system rankings produced by humans. However, our MIS measure and BLEURT have the highest correlations with human judgments. BLEURT performs best on this task because it correctly identifies the winner on 5 datasets out of 6. The popular measures BLEU, ChrF, and WMD identify the best system only on the xformal-FoST dataset.

Computational efficiency of the measures
While the correlation of measures with human judgments is important, the usability of the measure in real tasks can not be treated in isolation from its computational efficiency. The main capabilities of such measures are robustness and inference speed. One of the key functions of content preservation measures is to compare different TST or PG approaches with each other and ensure that different runs of the learning-based measure yield similar results. This problem does not apply to words or character n-grams-based models. However, this could yield some issues with trainable model-based measures. That is why it is crucial for all such measures to open-source trained weights. Moreover, when using such measures for comparison it is nec- essary to put the model into inference mode and freeze all layers. In such a case the model-based measures yield similar scores to similar text pairs regardless of the number of attempts or any hardware properties. Another blocker to the usage of a certain measure could be a long inference time. We conduct additional experiments by calculating the average inference time per sample for a subset of measures representing each class w.r.t. the average correlation of the measure on the task. We concatenate texts from both tasks into two united datasets. For trainable measures, we use a data loader with a batch size equal to eight. We load all trainable models to NVIDIA GeForce RTX 2080 Ti. All other measures are calculated sample-wise on Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz . We plot the results on Figure 6.
The most optimal measures are located at the bottom right corner of these plots, which means that the measure requires the least possible computational time and at the same time demonstrates a high correlation with human judgments. For the PG task, the MIS measure demonstrates the best performance and its average inference time is at the approximately same level as most of the other model-based measures. For TST task symmetric and asymmetric cross-encoders are the most optimal.

Conclusions
As our experiments show, encoder-based measures of content preservation correlate with human judgments much better than the traditional word or character-based measures such as BLEU on a wide range of datasets. In all paraphrase datasets and 9 out of 10 text style transfer datasets, the best-performing measures are based on the crossencoder or bi-encoder architecture.
We suggest a measure called MIS which is based on the idea that texts with similar meanings mutually entail each other. We show that the proposed architecture outperforms other measures in the evaluation of paraphrases and performs on par with the top-performing measures in the evaluation of text style transfer. More specifically, it is particularly successful in transferring between contemporary and old English and between formal and informal styles. Thus, we recommend using this measure for content preservation scoring for paraphrases and TST tasks in the aforementioned tasks and to use BLEURT for other TST tasks.
While the best measures in our analysis improve over the popular ones (e.g. BLEU) by a large margin, their correlation with human judgments is still far from perfect. We expect that even better measures of content preservation will be proposed in the nearest future. We also hope that the MIS measure and the performed large scale computational study could be applied to other NLP tasks, such as machine translation, text summarization, etc.