{'fraction_non_alphanumeric': 0.03856112127593468, 'fraction_numerical': 0.014637345945386128, 'mean_word_length': 4.988836055406243, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RuSentEval: Linguistic Source, Encoder Force!

The success of pre-trained transformer language models has brought a great deal of interest on how these models work, and what they learn about language. However, prior research in the field is mainly devoted to English, and little is known regarding other languages. To this end, we introduce RuSentEval, an enhanced set of 14 probing tasks for Russian, including ones that have not been explored yet. We apply a combination of complementary probing methods to explore the distribution of various linguistic properties in five multilingual transformers for two typologically contrasting languages – Russian and English. Our results provide intriguing findings that contradict the common understanding of how linguistic knowledge is represented, and demonstrate that some properties are learned in a similar manner despite the language differences.


Introduction
Transformer language models (Vaswani et al., 2017) have achieved state-of-the-art results on a wide range of NLP tasks in multiple languages, demonstrated strong performance in zero-shot cross-lingual transfer (Pires et al., 2019), and even surpassed human solvers in NLU benchmarks such as SuperGLUE (Wang et al., 2019). The success has stimulated research in how these models work, and what they acquire about language. The majority of the introspection techniques are based on the concept of probing tasks (Adi et al., 2016;Shi et al., 2016;Conneau et al., 2018) which allow analyzing what linguistic properties are encoded in the intermediate representations. A rich variety of tasks has been introduced so far, ranging from token-level and sub-sentence probing (Liu et al., 2019;Tenney et al., 2019) to sentence-level probing (Alt et al., 2020). A prominent method to explore the inner workings of the models involves training a lightweight classifier to solve a probing task over features produced by them, and assess their knowledge by the classifier's performance. Recently, the methods have been greatly extended to latent subclass learning (Michael et al., 2020), correlation similarity measures (Wu et al., 2020), information-theoretic probing (Voita and Titov, 2020), investigation of individual neurons (Durrani et al., 2020;Suau et al., 2020), and many more.
Despite growing interest in the field, English remains the focal point of prior research (Belinkov and Glass, 2019;Rogers et al., 2021) leaving other languages understudied. To this end, several monolingual and cross-lingual probing suites have been assembled (see Section 2), with a few of them following SentEval toolkit (Conneau et al., 2018;Conneau and Kiela, 2018). However, most of them directly apply an English-oriented method which is not guaranteed to be universal across languages, or use the Universal Dependencies (UD) Treebanks (Nivre et al., 2016) which tend to be inconsistent (Alzetta et al., 2017;de Marneffe et al., 2017;Droganova et al., 2018).
This work proposes RuSentEval, a probing suite for evaluation of sentence embeddings for the Russian language. We adapted the method for English (Conneau et al., 2018) to complement the peculiarities of Russian. In contrast to closely related datasets (Ravishankar et al., 2019;Eger et al., 2020), RuSentEval is fully guided by linguistic expertise, relies on annotations obtained with the current state-of-the-art model for Russian morphosyntactic analysis (Anastasyev, 2020), and includes tasks that have not been explored yet.
The contributions are summarized as three-fold. First, we present an enhanced set of 14 probing tasks for Russian, organized by the type of linguistic properties. Second, we carry out a series of probing experiments on two typologically different languages: English, which is an analytic Germanic language, and Russian, which is a fusional Slavic one. Keeping in mind that the English (Conneau et al., 2018) and Russian datasets are combined from different annotation schemas, we introspect five multilingual transformer-based encoders, including their distilled versions. We apply several probing methods to conduct the analysis from different perspectives, and support the findings with statistical significance. Besides, we establish countbased and neural-based baselines for the tasks. Finally, RuSentEval is publicly available 1 , and we hope it will be used for evaluation and interpretation of language models and sentence embeddings for Russian.

Related Work
Introspection of pre-trained language models for languages other than English, specifically Russian, is usually conducted in the cross-lingual setting. The primary goal of such experiments is to explore how particular linguistic properties are distributed in a given collection of multilingual embedding and language models. LINSPECTOR (Ş ahin et al., 2020) is one of the first probing suites that covers a wide range of linguistic phenomena in 24 languages. The benchmark uses UniMorph 2.0 (Kirov et al., 2018) to design the tasks since UD do not provide a sufficient amount of data for the considered languages. Despite this, UD Treebanks have become the main source for collection of multilingual probing tasks, limiting the scope to morphology and syntax (Krasnowska-Kieraś and Wróblewska, 2019). Other sources for the assembly include multilingual datasets created by means of machine translation, such as XNLI (Conneau et al., 2020b), or datasets that labelled with similar annotation schemes for named entity recognition (NER) and semantic role labelling (SRL) tasks (Ş ahin et al., 2020).
A few prior works (Ravishankar et al., 2019;Eger et al., 2020) that follow SentEval toolkit (Conneau and Kiela, 2018) directly apply the method designed for English to multiple typologically diverse languages, which raises doubts if such strategy is universal across languages that exhibit unique peculiarities, e.g. a free word order and rich inflectional morphology. This can lead to low quality of the datasets and unreliable experimental results, particularly for Russian. Consider a few examples for (BShift) task in Russian: Fedra zatem povesilas', a Tesey uznal pravdu. "Phaedra later hanged herself, and Theseus unraveled the truth." (Ravishankar et al., 2019), and Shestoe -zanimatsya nado svoim obrazovaniem "The sixth point is that you to take care need of your education" (Eger et al., 2020). The sentences are labelled as positive, meaning that they exhibit incorrect word order. While the word order changes can lead to the syntax perturbations for English, both sentences are still acceptable in terms of syntax for Russian. Moreover, the dataset sizes tend to be inconsistent across languages due to using UD Treebanks which makes it difficult to compare the results (Eger et al., 2020).
Another line of research includes probing machine translation models (Mareček et al., 2020) over multiple languages, and probing for crosslingual similarity by utilizing paired sentences in mutually intelligible languages (Choenni and Shutova, 2020a,b). Last but not least, such benchmarks as XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020) allow evaluating the current state of language transferring methods.

Probing Tasks
Data The sentences for our probing tasks were extracted from the following publicly available resources: Russian Wikipedia articles 2 and news corpora such as Lenta.ru 3 and the news segment of Taiga corpus (Shavrina and Shapovalova, 2017). We used rusenttokenize library 4 , a rule-based sentence segmenter for Russian, to split texts into sentences. Each sentence was tokenized with spaCy Russian Tokenizer 5 . The sentences were filtered by the 5-to-25 token range and annotated with the current state-of-the-art model for joint morphosyntactic analysis in Russian (Anastasyev, 2020). Besides, we performed two additional preprocessing steps. (1) We computed the IPM frequency of each sentence using the New Frequency Vocabulary of Russian Words (Lyashevskaya and Sharov, 2009) to control the word frequency distribution. The IPM values of each token lemma in the sentence (if present in the vocabulary) were averaged over a total number of token lemmas. The sentences with the IPM frequency of lower than 0.9 were filtered out. This allows discarding the majority of sentences that contain rare words, acronyms, abbreviations, or loanwords. (2) Syncretism is peculiar to fusional languages, that is, when a word can belong to multiple part-of-speech tags, or express multiple ambiguous morphosyntactic features (Baerman, 2007). Following (Ş ahin et al., 2020), we removed sentences in the semantic tasks (described below) where the target word has multiple part-of-speech tags. This step allows simplifying the probe interpretation and ensuring a fairer experimental setup in terms of the language comparison.
The total number of the annotated sentences after filtering is 3.6 mln, and they are publicly available. Each task consists of a 100k-sentence training set and 10k-sentence validation and test sets. There is no sentence overlap across the splits, and all sets are balanced by the number of instances per target class.
Surface properties tasks test if it is possible to recover information about surface properties from the contextualized representations. (SentLen) is a 6-way classification task aimed to predict a number of tokens given a sentence representation. Similar to (Adi et al., 2016;Conneau et al., 2018), we grouped sentences into 6 equal-width bins by length. The word content (WC) task tests if the information on the original words in a sentence can be inferred from its representation. We selected 1k lemmas from the source corpus vocabulary within the 1.5k-3k rank range when sorted by frequency, and sampled equal numbers of sentences that contain only one of these lemmas. The task is treated as a 1k-way classification that requires knowledge about lexical items and their inflectional paradigms.
Syntactic properties is a group of tasks that probe the encoder representations for syntactic properties. In the (ConjType) task, the sentences must be classified in terms of the type of connection between complex clauses. The objective of the classifier is to tell whether a sentence involves coordination or subordination.
(ImpersonalSent) is a binary classification task that aims to define if there is a lack of a grammatical subject in the main clause of a sentence. It is usually expressed by a singular third-person, reflexive, singular neuter, or invariable predicate form (smerkaetsya "it is getting dark"; zharko "it is hot"; pora idti "it is time to go"), an adverbial predicate phrase (bylo sovershenno tikho "it was absolutely quiet"), and intransitive verbs typically combined with a noun phrase in the instrumental (zapahlo rozami "it smells roses").
(TreeDepth) task tests whether the encoder representations store the information on the hierarchical and syntactic structure of sentences. Specifically, the goal is to probe for knowledge about the depth of the longest path from the root node to any leaf in the syntax tree. Similar to (Conneau et al., 2018), we obtained sentences where the tree depth and the sentence length are de-correlated. The tree depth values range from 5 to 9 which makes (TreeDepth) a 5-way classification task.
(Gapping) task deals with the detection of syntactic gapping that occurs in coordinated structures and elides a repeated predicate, typically from the second clause. We used data provided in the Shared Task on Automatic Gapping Resolution for Russian, or AGRR-2019(Ponomareva et al., 2019. For instance, the sentence Odin imel silu solntsa, drugoy -luny. "One had the power of the Sun, the other (had the power of) the Moon." comprises an omission of a repeating predicate in the non-initial clause with its semantics remaining expressed.
The N-gram shift task (NShift) is analogous to SentEval's (Bshift) that tests the encoder's sensitivity to incorrect word order. As opposed to English, only specific cases of word inversion in Russian lead to syntax perturbation. We, therefore, perturbed N-grams that correspond to a set of predefined morphosyntactic patterns. We used TF-IDF method from scikit-learn library (Pedregosa et al., 2011) to build an N-gram feature matrix that was further applied for the word order perturbation. For instance, we reversed adjacent words in prepositional phrases, numeral phrases, compound noun phrases, etc. Below is an example where the head of the prepositional phrase v schkolu 'to school' is inverted with the dependent noun: Segodnya on ne poshel shkolu v.
"He did not go school to today.' Semantic properties tasks rely on both syntactic and semantic structure of a sentence to recover a higher-level property. (SubjNumber) and (SubjGender) probe for the number and gender features of the subject in the main clause. Similarly, (ObjNumber) and (ObjGender) focus on the number and gender of direct object in the main clause. In the following tasks, the aim is to probe for the morphosyntactic features of the predicate or the head of a predicative construction of the main clause: predicate voice (PV), predicate aspect (PA), and predicate tense (PT). The latter is analogous to SentEval's (Tense) task.
The semantic tasks test if the contextualized representations not only capture the morphosyntactic features but also encode higher-level, structural and syntactic-semantic information (namely, the syntax tree hierarchy and the actant structure of a predicate). Note that the boundary between the surface, syntactic and semantic tasks is relatively blurred.

Methods
Probing Classifiers We trained linear and nonlinear classifiers over intermediate representations produced by the encoders 6 , using categorical crossentropy loss, and Adam optimizer (Kingma and Ba, 2015). For the non-linear classifier (MLP), we used the Sigmoid activation function, the number of hidden states of 250, and the dropout rate of 0.2. Training is run over 5 iterations with the L2regularization parameter ∈ [0.1, ..., 1e −5 ] tuned on the validation set, and the best classifier selected. The performance is evaluated by accuracy score.

Individual Neuron Analysis
Neuron-level introspection technique (Durrani et al., 2020) allows identifying top neurons that contribute most to a probing task and observe how these neurons are distributed across layers of the encoder. Similarly, we trained a linear probing classifier over concatenated sentence representations and used the weights to measure the importance of each neuron. The classifier is trained using Elastic-net regularization with L1 and L2 λ's ∈ [0.1, . . . , 1e −5 ] tuned on the validation set. Refer to (Dalvi et al., 2019;Durrani et al., 2020) for more details.
Correlation Analysis Correlation-based analysis techniques posed in (Wu et al., 2020) allow measuring similarity of the encoder intermediate representations without any linguistic annotation. We apply neuron-level (maxcorr) and representationlevel (lincka) similarity measures to investigate the encoders. maxcorr identifies pairs of neurons of the maximum correlation from two different layers. It is high when two layers have pairs of neurons with similar behavior. lincka provides a comparison of representations from different layers in a given collection of models. Two layers are assigned a high similarity if their representations behave similarly over all the neurons.

Baselines
We established a number of count-based and noncontextualized baseline features to train the probing classifiers as outlined in Section 4.2. We used Ngram range ∈ [1, 4] and top-150k features in the vocabularies for each count-based baseline. The count-based features include TF-IDF over character N-grams, TF-IDF over BPE tokens (Sennrich et al., 2016), and TF-IDF over Sentence-Piece tokens (Kudo and Richardson, 2018). We applied multilingual BertTokenizer and XLMRober-taTokenizer by HuggingFace to segment sentences into BPE and SentencePiece tokens. The noncontextualized features are mean-pooled monolingual fastText sentence embeddings (Bojanowski et al., 2017). We used monolingual fastText models for English 7 and Russian. The latter was trained

Results
How is the linguistic knowledge of two contrasting languages distributed in pre-trained multilingual encoders? This section describes the results for shared English and Russian probing tasks, covering surface properties (SentLen, WC), syntax (TreeDepth, NShift), and semantics (ObjNumber, SubjNumber, and Tense). We also report the results for the remaining tasks and the baselines in Appendix B. Table 1 presents the results of the linear classifier performance over the shared tasks. For the sake of space, we omit the results of the non-linear classifier and provide them in Appendix B. The best score for each task in each language is highlighted in grey, and the index number of the layer achieving the score is enclosed in square brackets. We observe that the surface and syntax tasks show similar trends for both languages, while a few semantic tasks reveal some differences. The overall pattern for the surface-level tasks (SentLen, WC) is that the probing curves 8 for both languages tend to be decaying after reaching the peak at the lower layers [1 − 4]. The exception is provided by M-BART which keeps the surface properties across the majority of layers. The baselines mostly perform on par with one another, with fastText and TF-IDF Char receiving the best score (see Appendix B).

Layer-wise Supervised Probing
(Nshift, TreeDepth) tasks demonstrate interesting distinctive features of LABSE and M-BART encoders with respect to the syntactic properties. Figure 1 depicts the probing curves over (Nshift) and (Bshift) tasks. First, LABSE is at most sensitive to the incorrect word order at the lower-tomiddle layers [2 − 8], as opposed to other encoders which distribute the information at the middle-tohigher layers [8 − 12]. Second, M-BART shows confident knowledge in TreeDepth task for both languages at the middle-to-higher layers [7 − 12] and reaches the peak at layer 12, in contrast to the rest of the encoders which generally distribute the property at the middle layers [5 − 7] (see Appendix B). The baselines performance varies from achieving a low score (TreeDepth) to being close to random choice (Nshift).
The probing curves for (PT) and (Tense) tasks illustrate that the models encode the property in a very similar fashion, achieving the peak score at the middle layers [4,5], and flattening the curves until the output layer (see Appendix B). In contrast, the behavior of the encoders on (SubjNumber) and (ObjNumber) is slightly different. The number of the subject for English is predominantly distributed at the middle-to-higher levels [5 − 12], while for Russian it is either at a steady pace from the very first layer (M-BART, M-BERT), or decaying at the lower and middle layers [3 − 5], and further increasing close to the higher levels [8−12] (LABSE, MiniLM). Other differences are found in the results for (ObjNumber) task. Specifically, the number of direct object for English is best inferred at the lower layers of LABSE [2, 3], as compared to the middle-to-higher layers of other encoders [6 − 9]. Despite this, the property is similarly distributed across the layers of the models. In the same manner as for English, LABSE concentrates the knowledge for the Russian task in layer 2 but decays once reaching the peak. However, the other models exhibit a distinct behavior as well, with the property best encoded at the middle layers [6, 7] (M-BERT, M-BART), or at layer 10 (XLM-R, MiniLM). Notably, the baselines receive a strong performance over the majority of the semantic tasks for both languages, meaning that they can be solved using sub-word features that can capture lexical, or morphosyntactic information.
Bootstrap We estimate statistical significance of the supervised probing by means of layer-wise bootstrap procedure (Berg-Kirkpatrick et al., 2012). Typically, we observe that when the probing curve rises, the layer-wise difference is statistically sig-nificant, while at the peak of the probing curve it turns insignificant. We can treat these observations in the following way: starting from the peak of the probing curve, the models acquire the knowledge needed for the task, and do not generally lose it in the higher layers.

Neuron-level Analysis
In contrast to layer-wise supervised probing (Section 5.1) which introspects each layer independently, individual neuron analysis allows exploring the distribution of top neurons selected from the entire encoder. This provides an alternative perspective to which layers contribute predominantly towards the probing tasks. Note that we trained distinct classifiers regularized with Elastic-net (Zou and Hastie, 2005) to estimate the importance of the neurons 9 . Figure 2 presents the results for (SentLen) tasks in Russian and English. Different from other models which predominantly capture the property by neurons at the lower layers [1 − 4], M-BART distributes the information across all the layers. Let us analyze the results with respect to some syntactic and semantic levels. A similar behavior of the encoders by language is observed over (NShift) and (TreeDepth) tasks (see Appendix C). M-BERT, XLM-R, and MiniLM generally capture the sensitivity to illegal word order by neurons at the middle and higher layers, while it is contributed by fewer layers of LABSE (Ru: [2 − 4]; En: [4 − 7]), and M-BART which surprisingly stores the property at layer 12 for each language. Another interesting finding is that the depth of the syntax tree is typically spread across most of the layers of each encoder. An exception to this pattern is M-BART which locates the knowledge at the middle-to-higher layers [7 − 11].
The neuron distributions for (ObjNumber) task are akin by language and slightly different by encoder, showing that the number of the direct object is learned by at least 5-7 layers of different encoder depth. At the same time, the properties for (SubjNumber) and (PT, Tense) tasks are captured differently by the encoders. M-BERT, XLM-R and MiniLM reveal similar behavior by task, whereas LABSE specializes the knowledge at either the lower layers (Ru, SubjNumber; and En, Tense: [1 − 4]), or the higher ones (En, SubjNumber; and Ru, PT: [10 − 12]). On the other hand, the most contributing neurons of M-BART are predominantly spread across all the layers.

Correlation Analysis
For analyzing the encoders by means of the correlation-based techniques, we used 1k stratified sentences from each test set of the shared tasks. We obtained the sentence representations and computed the measures applying the publicly-available code (Wu et al., 2020). Figure 3 shows heatmaps of similarities between layers of the encoders under neuron-level and representation-level measures for English. Notably, the heatmaps are very alike to the ones for Russian, which we enclose in Appendix D. maxcorr (Figure 3a

Discussion
Three encoders exhibit similar behavior, but two other differ from them in capturing linguistic properties The most striking distinction based upon the results is that M-BART and LABSE exhibit different behavior for both languages, as opposed to M-BERT, XLM-R, and MiniLM (Section 5.1, 5.2). Specifically, M-BART generally tends to distribute the surface properties across all the layers, unlike other encoders where the information is specialized by the lower ones. The syntactic properties tend to be localized at the higher layers [11 − 12] which is as well demonstrated in other Russian tasks (Gapping, ImpersonalSent). In contrast to other models that capture the semantic properties at the middle-to-higher layers, LABSE typically displays the knowledge at the lower-to-middle layers. The analysis of the understudied encoders contradicts the common findings on transformer models that syntactic information is stored at the middle layers, while semantic knowledge is most prominent at the higher layers (Rogers et al., 2021). Therefore, there is still room for exploring transformer-based models that differ by architecture type, or by a set of pre-training objectives. MiniLM inherits linguistic properties from M-BERT Teacher models are usually compared with their distilled versions on a range of downstream tasks (Tsai et al., 2019), and little is explored on what language properties they bequeath to their students. We find that MiniLM is likely to mimic the behavior of XLM-R rather than of M-BERT, most probably due to using the same tokenization method. The similarity is most demonstrated by the individual neuron analysis (see Appendix C) and lincka (see Figure 3b). Along with that, the model receives comparative performance under layer-wise supervised probing (see Appendix B).
Surface and syntactic information is learned in a similar manner The probing curves demonstrate that the surface and semantic properties of the two languages are similarly distributed in the encoders. The surface properties are generally captured at the lower layers, and the pattern of the curves is decaying towards the output layer. The syntactic properties are predominantly inferred at the middle or higher layers, and the semantic tasks reveal a number of differences described in Section 5. Notably, the results obtained under layer-wise supervised probing (Section 5.1) are supported by the individual neuron analysis (Section 5.2), and the representation-level analysis (Section 5.3).

Encoders may have similar distributions of neurons by task, but different individual neurons
Two neuron-level introspection methods allow drawing the following finding. Despite that topneuron distributions in the encoder layers share similar patterns in the majority of probing tasks (Section 5.2), maxcorr induces high intra-encoder similarities and low inter-encoder similarities (Section 5.3). In other words, the neurons can similarly localize particular properties in the layers and yet behave differently across the models.

Conclusion
This paper introduces RuSentEval, an enhanced probing suite of 14 probing tasks that cover various linguistic phenomena of the Russian language. We explored five multilingual transformer encoders over the probing tasks on two typologically contrasting languages -Russian and English. The experiments are conducted using a combination of complementary probing methods, including layer-wise supervised probing, individual neuronanalysis, neuron-and representation-level similarity measures. Particularly, the behavior of the encoders under probing classifiers is reflected in distributions of top neurons with respect to a task, and the similarity is supported by linear centered kernel alignment method. We found that despite the language distinctions, the surface and syntactic properties are learned in a fairly similar manner, and the semantic knowledge is captured differently in a majority of tasks. We believe that the findings make the ongoing studies on cross-lingual transfer even more promising, specifically from English to Russian or vice versa. In contrast to prior works on how linguistic knowledge is represented in transformers, the analysis of the understudied models reveals that syntax and semantics can be differently represented across the layers. Besides, we found that different encoders often have similar distributions of neurons that contribute most to a probing task, and yet differ under neuron-level similarities. We also observed that distilled models inherit linguistic properties from their teachers, and receive comparative performance on a number of probing tasks. An exciting direction for future work is to investigate the correlation between probing and high-level downstream tasks, in order to identify which linguistic properties anticipate the behavior of a model in action.

PA
On nikak ne ob"yasnil svoyu pozitsiyu .  The results reported in the main body of the paper are obtained with Logistic Regression classifier over the shared tasks (see Section 5.1). We present detailed results for both linear and non-linear classifiers on all Russian and English tasks in Tables 3-6. Tables 7-9 show the results of the baselines.

B.2 Probing Trajectories
Figures 4-9 show the probing curves of Logistic Regression classifier over the shared tasks (see Section 5.1). The results obtained with MLP classifier are pretty consistent with the ones presented in this Appendix. and En is at the right. X-axis=Layer number, Y-axis=Accuracy score. En is at the right. X-axis=Layer number, Y-axis=Accuracy score. and En is at the right. X-axis=Layer number, Y-axis=Accuracy score. and En is at the right. X-axis=Layer number, Y-axis=Accuracy score. and En (Tense) is at the right. X-axis=Layer number, Y-axis=Accuracy score. Figure 9: The probing results of Logistic Regression classifier for each encoder on WC. Ru is at the left; and En is at the right. X-axis=Layer number, Y-axis=Accuracy score.