{'fraction_non_alphanumeric': 0.026313098605951123, 'fraction_numerical': 0.015235693104741129, 'mean_word_length': 4.7665094339622645, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CLISTER : A Corpus for Semantic Textual Similarity in French Clinical Narratives

Modern Natural Language Processing relies on the availability of annotated corpora for training and evaluating models. Such resources are scarce, especially for specialized domains in languages other than English. In particular, there are very few resources for semantic similarity in the clinical domain in French. This can be useful for many biomedical natural language processing applications, including text generation. We introduce a definition of similarity that is guided by clinical facts and apply it to the development of a new French corpus of 1,000 sentence pairs manually annotated according to similarity scores. This new sentence similarity corpus is made freely available to the community. We further evaluate the corpus through experiments of automatic similarity measurement. We show that a model of sentence embeddings can capture similarity with state-of-the-art performance on the DEFT STS shared task evaluation data set (Spearman=0.8343). We also show that the corpus is complementary to DEFT STS.


Introduction
Semantic Textual Similarity (STS) is a Natural Language Processing (NLP) task aiming at evaluating the proximity between two pieces of text. It has various applications, such as question answering or text summarization, and has its roots in earlier work about paraphrase (Dolan et al., 2004) and textual entailment (Dagan et al., 2010). It is a well-known NLP task that has been regularly studied through evaluation campaigns like SemEval since 2007 and challenges like the 2020 edition of the French challenge DEFT (Défi Fouilles de Textes) . Work on the automatic detection of sentence similarity is supported by resources such as the STS Benchmark dataset (Cer et al., 2017) and the SICK dataset (Marelli et al., 2014), which include pairs of sentences annotated with a degree of similarity (STS Benchmark) or a relation label (SICK dataset). To our knowledge, there are no existing STS-oriented corpora in French except the French Corpus for Semantic Similarity  from the DEFT challenge we mentioned, and no STS corpora in the medical domain. Moreover, the notion of similarity is hard to define and existing corpora are not necessarily accompanied with guidelines providing an exact definition of the notion. In English, several STS corpora in the medical domain exist. The BIOSSES corpus (Sogancıoglu et al., 2017) contains 100 sentence pairs from the Text Analysis Conference track on biomedical summarization in English. Criteria for similarity are defined in this corpus but rely on the knowledge of the annotators for identifying information that can be considered as "impor-tant" or as "details". The sentence pairs are annotated with a similarity score on a scale going from 0 to 4. The larger version of the MedSTS Corpus  used in the n2c2/OHNLP shared task  contains 2,054 medical sentence pairs in English annotated by two annotators who are experts in the medical domain. The similarity criteria in this corpus are defined following the ideas used for the BIOSSES corpus, but adding an extra degree on the similarity scale (0 to 5). The French Corpus for Semantic Similarity  used in the 2020 edition of DEFT contains 1,010 annotated sentence pairs taken from the CLEAR corpus (Grabar and Cardon, 2018). The definition of the similarity throughout the annotation process of this corpus relied on the intuition of the annotators. The work presented herein shows that annotating a sentence similarity corpus in a technical domain is a hard task, especially when annotators have variable levels of knowledge of the domain. Defining precise criteria to give a definition of similarity specific to the domain can significantly improve inter-annotator agreement. The main contributions of this work are the following: • we introduce a definition of similarity guided by linguistic and clinical criteria; • we propose a new STS corpus, called CLISTER, composed of 1,000 sentence pairs from the clinical domain; • we evaluate the newly created corpus for semantic similarity using a state-of-the-art sentence embedding model. • we compare the corpus with an existing STS corpus from French encyclopedias by experimenting with other semantic similarity tests on both corpora to test their compatibility.

Computing Similarity
Lithgow-Serrano et al. (2019) and Lara-Clares et al. (2021) review corpora and methods for computing sentence similarity in English. In this section, we briefly introduce elements that are most relevant to our work. Working on similarity implies finding methods to compare the objects that are involved. For sentences, some methods can be applied directly on two strings, like the Levenshtein distance (Levenshtein, 1966).
Other methods aim at encoding the sentence into a vector of numbers and then use existing metrics that work efficiently on such vectors. A basic method to transform text sequences into vectors of numbers is to compute the TF-IDF of all the words within the sentence, using the vocabulary of the entire corpus.
In order to obtain a sentence representation that retains (some) semantic information, deep learning models can be trained to compute sentence embeddings. In our work, we used the Sentence-BERT model (Reimers and Gurevych, 2019). Based on the BERT language model (Devlin et al., 2019), the architecture of Sentence-BERT allows for adjustments of the BERT output in order to generate fixed-sized sentence embeddings. Once the embeddings are obtained, a similarity score between the vectors can be computed using cosine similarity.

Source Corpus
For this work, we used the CAS corpus , a French medical corpus containing clinical case descriptions. Clinical cases cover a variety of clinical information such as descriptions of the medical history of the patients, as well as treatments or follow-ups. These descriptions can apply to any medical disorder. The data in the corpus has been de-identified and the publication of documents is done with the written permission of patients. The corpus can be obtained from the DEFT 2020 shared task organizers 1 . The CAS corpus has also been linguistically and semantically annotated, but we did not use those annotations in our work. We chose to use this corpus as the basis for our work because it is one of the largest medical corpora available in French, containing data that is intended for the medical community and not simplified to help external understanding.

Selecting Sentence Pairs
The corpus was split into sentences using the Talismane parser (Urieli, 2013) for French, with default parameters. We made the hypothesis that two randomly selected sentences are unlikely to be semantically related.
1 https://deft.limsi.fr/2020/ That means that we needed to find a way to filter sentence pairs so that the resulting corpus is not composed of a high majority of unrelated sentence pairs. We drew our inspiration for the selection of sentence pairs from , to which we added a filter. In this article, sentence pairs are selected using three metrics: the Ratcliff/Obershelp pattern-matching algorithm (Black, 2021) and the Levenshtein distance at character level, and the cosine similarity at token level. The mean score of the three metrics is computed and the sentence pairs with a mean score above a threshold of 0.45 are selected. For our work, we thought it would be sufficient to keep one method at character level and one method at token level; so we only used the Levenshtein distance and the cosine similarity. We used the same threshold of 0.45 for the mean score of the two metrics to select sentence pairs. First, we encoded our sentences according to a Bagof-Words representation with a TF-IDF weighting scheme. We then used the cosine similarity to keep sentence pairs above a given threshold of 0.4, slightly smaller than our final threshold (0.45) in order to filter a good number of sentence pairs before the Levenshtein distance needs to be computed, which is a lot more costly than computing cosine similarity. Then, we kept only the pairs where the two sentences were close in number of tokens to exclude sentence pairs where a small sentence is included in a larger one and where the similarity of the remaining non-common part would be hard to evaluate. We then computed the Levenshtein distance between the sentences of a pair. In the end, we kept the sentence pairs where the mean of the cosine similarity and the Levenshtein distance was above a threshold of 0.45, as was done in .

Defining Criteria
We defined similarity by decomposing it into three linguistically and clinically driven dimensions. This helps the annotators focus on those aspects and therefore be more consistent. We divided the notion of similarity into three categories. We present here those categories, from the less important to the most important. Examples can be found in section 3.3.2. Surface similarity The surface similarity concerns the structural similarity. This similarity is based on grammatical words or words that are not related to the domain. Two sentences that have a surface similarity can be syntactically close but semantically distant. Semantic similarity In our corpus, semantic similarity concerns medical concepts. The closer the concepts are to one another, the higher the similarity. These concepts can refer to medications, diseases, procedures, and others. Clinical compatibility Going further into the semantics, clinical compatibility is an assessment of whether sentences in a pair can refer to the same clinical case.

Scoring scale with examples
When annotating sentence pairs with a similarity score, it is common to use scores going from 0 for the minimum similarity score to 5 for the maximum score, as in the STS Benchmark (Cer et al., 2017) and MedSTS . We decided to use this scale for our corpus, using only round values. Each score from 0 to 5 is defined as follows 2 : Similarity score 0: For sentence pairs with only surface similarity, such as words non-specific to the medical domain or stop-words.
(1) a. Il n'y avait pas de résidu post-mictionnel. [There was no post-void residual urine.] b. Il n'avait pas de facteurs de risque cardiovasculaire notable. [There were no notable cardiovascular risk factors.] Both sentences in (1) have a similar structure, starting with "There was / were no...". However the remaining parts of the sentences are absolutely not related. Similarity score 1: For sentence pairs with only surface similarity, concerning at most one medical entity.
(2) a. L'examen physique révélait une légère sensibilité de la fosse lombaire droite. Both sentences (2) share a common structure, with a certain type of medical exam revealing a symptom. Besides this structure, the two sentences are not related. Similarity score 2: For sentence pairs containing medical concepts with low semantic similarity, but no clinical compatibility. Typically, sentences in a pair can concern a disease, a procedure, or a drug.
[Radionuclide scan was negative.] Both sentences in (3) are about a scan, which is different from one sentence to another but ends up with a common diagnosis. This example also shows the potential difficulties of annotating in a technical domain, because judging the proximity of those two scans is not necessarily obvious. Similarity score 3: For sentence pairs with semantic similarity on several medical concepts making them partially clinically compatible.
Similarity score 4: For sentence pairs with high semantic similarity and clinical compatibility. One sentence may contain more information than the other may, and vice-versa. The sentences in (5) describe a similar situation and are clinically compatible. The main difference is the timing of the follow-up. There is also more information in sentence (5-a) with the sex of the patient (in French) and "disease free" is also more specific than "in good health". Even if sentence (5-b) does not present this information, there is no contradiction between them.
Similarity score 5: For sentence pairs with high semantic similarity and full clinical compatibility. The sentences have globally the same meaning, while one may be more specific than the other. Here, we differentiate between being more specific and containing more information. The sentences in (6) present high similarity. The only difference is the added precision on the tumor markers in sentence (6-a), but the sentences are equivalent.

Balancing Categories in the Corpus
Having a closer look at the annotations of the 600 sentence pairs selected initially (see Section 4), we observed that there were only a few sentence pairs annotated with extreme similarity scores of 0 and 5. However, we think it is important to have a fair representation of those extremes in order to have a reliable definition of similarity. We also aimed to increase the size of the corpus to a scale that would allow for the training and testing of statistical similarity methods. For those reasons, we decided to expand the corpus at lower cost by semi-automatically adding sentence pairs with scores of 0, 4, and 5. Sentence pairs with a similarity score of 0 are the easiest to get, given the hypothesis that taking two random sentences in the corpus, they are very unlikely to be similar. We thus collected 210 random sentence pairs from the corpus, with the same length constraint that we used before, and annotated them with the score 0. To make sure that no similar sentence pairs are remaining, we computed the cosine similarity between the sentence embeddings obtained with SENTENCE-BERT (Reimers and Gurevych, 2019) for each pair. The highest similarity score for a pair was 0.453. From some experience working with SENTENCE-BERT, sentence pairs with a similarity score below 0.5 are already very unlikely to have any similarity besides surface similarity. We manually checked the sentences with a cosine similarity score above 0.3. About 15 pairs were reviewed, leading to the substitution of 2 pairs. Sentence pairs with high similarity were selected using the FAISS library (Johnson et al., 2021). Sentence embeddings were computed for all sentences in the corpus, and a similarity matrix was created. We kept the highest-ranked pairs and we manually annotated pairs with a similarity score of 4 or 5, while removing the pairs that were not similar enough for the annotator's criteria. This process was done until 190 new pairs were retrieved.

Manual annotation of semantic textual similarity in French clinical text
The four authors of the paper annotated the corpus. We will name them here A1, A2, A3, and A4 3 . A1 has experience in the annotation of biomedical text, A2 in semantic representation, A3 is an early-stage researcher in NLP, and A4 has experience in the creation of annotated resources.
We led the first round of annotation where three annotators (A1, A2, and A3) annotated a sample of 100 sentence pairs without any discussion between them or guidelines. The goal of this first annotation step was to test the intuition of the annotators and get an idea of the difficulty of the annotation task. Once annotated, this first sample was used as support for discussion and the conception of the annotation 3 Not according to authors' order guidelines. The final annotation of the sentence pairs of this sample was obtained from a global consensus between the annotators. A second sample of 100 sentence pairs was then annotated. The three original annotators were joined by the fourth one (A4), who did not participate in the discussion during the first sample. Besides the additional set of annotations, this new contribution was useful to assess the quality of the annotation guidelines, which should be understandable to annotators other than the authors. Inter-annotator agreement was computed for the second sample. It was found substantial, so each annotator was given another sample of 100 sentence pairs. In total, 600 sentence pairs were manually annotated.

Pilot annotation
We present herein the results of the annotation of the first sample of 100 sentence pairs, which was done without discussion between annotators or guidelines. For the inter-annotator agreement, we computed Krippendorff's α (Krippendorff, 2013) using a Python library implementing the metric 4 with interval data type and default parameters. The α for this first sample was 0.239. This poor value shows a significant disagreement between the annotators. Figure 1 displays the number of sentence pairs for each score and each annotator. We can observe an important heterogeneity of the distribution for each annotator. In this sample, annotator 2 is by far the strictest of the three annotators, with more than 60 sentence pairs annotated with the minimum score 0 and an overall average score of 0.64. Annotators 1 and 3 are more balanced between scores but annotator 3 presents a much higher average score, 2.74, against 1.99 for annotator 1. The high disparity in the first sample annotation shows that annotating clinical data with similarity based on the annotators' intuition is not trivial. Annotators can have a very distinct perception of the concept of simi-larity. This is why during the discussion and the elaboration of the annotation guide, our goal was to define precise criteria to judge similarity in a clinical context. The annotators discussed each sentence pairs of the first sample and agreed on a consensual score. The resulting distribution is displayed in Figure 2. The small number of pairs annotated with the lowest score of 0 probably means that the criteria we used to select candidate sentence pairs are often enough to achieve some similarity regarding the criteria we defined. At the same time, the fact that there is only one sentence pair annotated with the highest score of 5 probably means that such a degree of similarity between two sentences is not very common using our criteria.

Annotation consolidation
The results of the annotation of the second sample of 100 sentence pairs are shown in Figure 3. In this figure, annotators A1, A2, and A3 are the same as in sample 1, while annotator A4 is the new one. Overall, we can see a similar trend between Figure 3 and Figure 2, with a small number of sentence pairs annotated with the highest and lowest scores, 5 and 0, and most pairs in between. Only annotator A2 annotated an important number of sentence pairs with the score of 0. On this sample, we obtained a Krippendorff's α of 0.686, which is much better than the α achieved on the first sample and seems satisfactory considering the complexity of the corpus 5 . For the final annotation of this sample, we decided to compute the mean similarity value for each pair. In the end, the mean score for this sample is 2.40 (±1.17). This mean score and the relatively small standard deviation show that most sentence pairs are annotated with scores between 1 and 4, therefore that the filters we used when selecting sentence pairs often ensure at least a small similarity in our criteria and that highly similar sentence pairs are rare. We considered that the results obtained on the second sample were good enough to let the annotators independently annotate a new sample of 100 each. Table 1 shows the mean scores for the four new samples, which were annotated by a (different) single annotator. We can see that annotator A2 obtained a mean score very close to the mean score obtained on the second sample. The mean scores for annotators A1 and A4 are not very far, with a difference of approximately 0.40. Lastly, annotator A3 obtained a slightly lower score of 1.89. In general, mean scores are not very far from the tendency we observed in the previous sample.

Global Statistics
The final corpus of 1,000 annotated sentence pairs, with an average length of 15.34 tokens (±9.2) per sentence, contains 30,942 tokens, including punctuation marks. Those numbers were computed from the TAL-ISMANE parser output. Figure 4 shows the number of sentence pairs for each label in the corpus. With the semi-automatic selection of sentence pairs representing high/low similarity, the highest and lowest scores are now prevalent.

Evaluation Settings
In order to extrinsically evaluate the similarity defined in our corpus, we decided to test whether a model /DEHO 3URSRUWLRQ Figure 4: Proportion and number of sentence pairs for each similarity score in the final corpus (1,000 pairs).
trained on our data can account for our definition of similarity. For this experience, we used the sentence embedding model SENTENCE-BERT to build the representation of each sentence and predict the similarity between each pair of sentences using cosine similarity between the two resulting embeddings.
To perform this evaluation, we split our 1,000 sentence pairs into training and testing sets. We chose 600 sentence pairs for training and 400 sentence pairs for testing. Since we added a substantial amount of pairs semiautomatically, we made sure to have our full manually validated first sample in the test data. This ensures that the test set comprises the most reliable gold standard obtained through consensus annotations. The rest of the data was split randomly while controlling that label distribution was consistent in the train and test partitions. As mentioned above, experiments were conducted using SENTENCE-BERT, a pretrained multilingual model 6 covering 15 languages 7 , including French (Reimers and Gurevych, 2020). We used the basic training function of SENTENCE-BERT 8 by varying the number of epochs and the number of warm-up steps. We used default values for the rest of the parameters. Overall, the best results of the model were achieved with 5 epochs and 10 warm-up steps. The results we present in this paper are obtained with those parameters when fine-tuning SENTENCE-BERT. In order to get robust results, all experiments were repeated three times with shuffled training data. Therefore, we present here the mean values achieved with each configuration. Two metrics were used for evaluation: Spearman cor-relation and EDRM (Exactitude en Distance Relativè a la Solution Moyenne or "Accuracy as Relative Distance to Mean Solution"). EDRM is a metric created for the evaluation of the DEFT task for nuanced assessment of sentence similarity. This metric measures the mean distance between the true similarity scores and the predicted scores, taking into account the maximum distance possible from the true scores for each prediction. For example, if the true score is 0, the maximum distance is by predicting the score 5. The maximum distance here is 5 (5 -0 = 5). If the true score is 2, the maximum distance is also achieved by predicting the score 0. However, the maximum distance here will be 3 (5 -2 = 3). The formula for computing EDRM on a set of n sentence pairs, where h is the prediction and r is the reference score, is shown in equation 1.
Spearman's correlation is one of the standard metrics used to evaluate STS tasks. Both measures give a coefficient describing how much the variations of two variables are related, the variables here being the expected similarity score and the predicted similarity score. Table 2 presents the results of evaluating the similarity on our entire test set. When the model is used without fine-tuning, it achieves acceptable scores for EDRM (0.7149) and Spearman's correlation (0.7547). We observe a significant improvement for both EDRM and Spearman's correlation when the model is fine-tuned on the training data (+0.1261 for EDRM and +0.1123 for Spearman's correlation). It is encouraging to see that the model could adapt to an extent to our definition of similarity. However, the use of SENTENCE-BERT in the process of selecting high-scoring sentence pairs added to the  Table 2: Results for similarity experiments with different combinations of training and testing sets using SENTENCE-BERT pre-trained multilingual model (distiluse-base-multilingual-cased-v1). Training was done with 5 epochs and 10 warm-up steps.

Automatic semantic text similarity measurement in CLISTER
corpus could introduce a bias in the evaluation as the model's predictions will align with the reference for these sentence pairs. We also experimented using a bias-free subset of the test set, namely the sample of sentences that were independently annotated by three annotators followed by the creation of consensus annotations (CLIST ER 100 ). The results of the experiment are shown in Table 2.
The value for EDRM is significantly lower on this part of the testing data than on the whole test when the model is not fine-tuned (0.6323 against 0.7149), and Spearman's correlation drops drastically (0.3794 against 0.7547). When fine-tuning the model, the value for EDRM increases clearly (+0.1917) and is close to the value achieved on the whole test set. Spearman's correlation nearly doubles (+0.3546), but remains inferior to the value obtained on the complete test set. Those important improvements between the vanilla model and the fine-tuned model on this particular part of the test set show that the model was not necessarily familiar with our conception of similarity, but was able to learn from the training data to adapt to an extent, with a quite small amount of training data.

Automatic semantic text similarity measurement in CLISTER vs. DEFT STS
Cross corpus evaluations. We conducted cross corpus evaluations to assess the benefit of new data on STS performance.
To compare our corpus with the existing French STS corpus from the DEFT shared task, we experimented with the sentence embedding model SENTENCE-BERT on both datasets, crossing the training and test-ing data. We can also see in Table 2 that the DEFT STS test set gets lower results on average for the metrics we used. Several elements of our experiments show that the DEFT STS corpus and the CLISTER corpus we created are quite different. SENTENCE-BERT performed globally better for the STS task on the CLISTER corpus than it does on the DEFT STS corpus. Moreover, fine-tuning on one corpus while testing on the other decreases the performance of SENTENCE-BERT, and fine-tuning on both corpora gets slightly lower results than fine-tuning only on the corpus corresponding to the test set. The difference in performance and this noncompatibility between the two corpora can be related to the nature of the data (clinical for CLISTER, encyclopedic for DEFT STS) and/or to the definition of similarity underlying the similarity scores within the corpora.

Comparison to DEFT's results
The results of our experiments using DEFT STS for training and testing can be directly compared to those of systems submitted to the DEFT. The best perfor-mance in the task was obtained by a method representing sentence pairs by similarity features leveraging a wide range of similarity scores and training an ensemble classifier on this feature representation (Dramé et al., 2020). The method used in our experiments offers individual representation for sentences and yields a higher Spearman correlation (0.8343 vs. 0.7769) but lower EDRM (0.8217 vs. 0.7926).

Conclusion
We introduce in this paper a new STS corpus of 1,000 sentence pairs in the clinical domain in French. The annotation of the corpus followed a meticulous procedure with a good number of annotators (3 to 4). This procedure, combining discussions between annotators and the elaboration of a precise annotation guide, ensured the consistency of the annotation, with a correct inter-annotator agreement given the complexity of the corpus. We confirmed the quality of this corpus by experimenting with the assessment of sentence similarity using Sentence-BERT, a state-of-the-art sentence embedding model. We showed that a model trained on our data can capture our definition of similarity. We compared our corpus with the DEFT STS corpus, another STS corpus in French with nearly the same amount of annotated sentence pairs (1,010). We used once again Sentence-BERT to assess sentence similarity on both corpora, separately and jointly, varying the data used to train the model. We showed that the model would not adapt to the similarity of a corpus when trained on the other, and training on both corpora does not improve the results. We concluded that the two corpora are complementary, but we have yet to determine if this complementarity comes from the different definitions of similarity (intuition for DEFT STS, criteria for CLISTER), from the type of data (encyclopedia for DEFT STS, clinical cases for CLISTER), or both. The CLISTER corpus is freely available to the research community at https://gitlab.inria. fr/codeine/clister. The availability of an STS corpus in the clinical domain can help build models for information retrieval in the domain. In the future, we plan to use the CLISTER corpus in order to train a Sentence-BERT model and experiment with the retrieval of similar sentences in clinical corpus to assess the privacy of specific clinical sentences with respect to individual patients. It would also be interesting to further investigate the source of the differences between the DEFT STS and the CLISTER corpora.

Acknowledgements
This work was supported by the French National Agency for Research under grant CODEINE (artificial text COrpus DEsIgNed Ethically) ANR-20-CE23-0026-01.