{'fraction_non_alphanumeric': 0.03508464477235288, 'fraction_numerical': 0.016753706494689987, 'mean_word_length': 4.810997963340122, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Clause Topic Classification in German and English Standard Form Contracts

So-called standard form contracts, i.e. contracts that are drafted unilaterally by one party, like terms and conditions of online shops or terms of services of social networks, are cornerstones of our modern economy. Their processing is, therefore, of significant practical value. Often, the sheer size of these contracts allows the drafting party to hide unfavourable terms from the other party. In this paper, we compare different approaches for automatically classifying the topics of clauses in standard form contracts, based on a data-set of more than 6,000 clauses from more than 170 contracts, which we collected from German and English online shops and annotated based on a taxonomy of clause topics, that we developed together with legal experts. We will show that, in our comparison of seven approaches, from simple keyword matching to transformer language models, BERT performed best with an F1-score of up to 0.91, however much simpler and computationally cheaper models like logistic regression also achieved similarly good results of up to 0.87.


Introduction
So-called standard form contracts, i.e. contracts that are drafted unilaterally by one party of the contract, usually a company, like terms and conditions of online shops or terms of services of social networks, are cornerstones of our modern economy. While the concept of a contract that is completely decided upon by one party might seem unfair and inherently flawed, its existence is a necessity in our modern economy. It would simply not be possible for companies like Amazon, Facebook or Google, to negotiate individual contract terms with each of their customers.
It is largely acknowledged, that most consumers do not read such contracts before buying something online or registering for a service. The actual share of consumers that regularly read such contracts ranges from as little as 3.5% (Plaut and Bartlett III, 2012) to 9% (Braun, 2021) in the literature. In acknowledgement of this fact, lawmakers around the world have tightly restricted the provisions that can be made by standard form contracts in a bid to protect consumers. This makes them an interesting subject for different Natural Language Processing (NLP) tasks, because they have a high economical and therefore practical relevance, but are still somewhat restricted with regard to their content. In this work, we use contracts that have been drafted under the jurisdiction of the European Union (EU). Many of the regulations applying to standard form contracts in this jurisdiction originate from the Council Directive 93/13/EEC of 5 April 1993 on unfair terms in consumer contracts.
While standard form contracts are relevant in almost all kinds of business-to-consumer transactions, like banking, insurances, and data processing, we here focus on Terms and Conditions (T&C) from online shops for three reasons: They have high economical relevance, they are publicly available in large quantities on the internet in a machinereadable format, and they are among the contracts which are least likely to be read, compared to a contract for life insurance, for example.
Being able to automatically classify the topics of clauses in T&C could help consumers to find relevant regulations faster and make more informed decisions, but it could also support legal professionals, like lawyers specialising in consumer protection law, in their work.

Related Work
Contract review is one of the main commercial applications of NLP in the legal domain (Dale, 2019). Unlike standard form contracts, "normal" contracts, i.e. contracts that have been negotiated by all contracting parties, allow for more variation and are less regulated, especially in business-to-business contexts. Therefore, building a taxonomy of clause topics and performing topic classification (i.e. a supervised approach), would be less suitable for such contracts.
For reasons of data availability, most research projects use publicly available contracts, which happen to be standard form contracts, like T&C from online shops, Terms of Services (ToS) from online platforms, and, since the introduction of the General Data Protection Regulation (GDPR) in the EU, one particular focus has been on privacy policies. Although, legally, it is not settled whether they have a contractual status or not (Raysman and Brown, 2010), from an NLP perspective, they can be treated as a special variety of standard form contracts.
Most of the existing research on the analysis of standard form contracts is focusing on automatically finding void clauses. The CLAUDETTE project , for example, focuses on finding so-called "unfair clauses", which are void under EU legislation, in ToS from large online platforms like Facebook or Netflix. Later, they also applied their approach to privacy policies (Liepina et al., 2019). Similarly, Braun and Matthes (2021) focus on finding void clauses in T&C from online shops by using a fine-tuned BERT model, in earlier work, they summarised specific aspects of T&C using an abstractive summarisation approach (Braun et al., 2017). In comparison to these works, which try to make a fully automated decision on the validity of clauses, the classification of clause topics could be used to support humans in the decisionmaking process, by helping them to find relevant clauses faster.
In the area of privacy policies, approaches are more diverse. Ravichander et al. (2019), for example, presented a Q&A system that can answer user questions about privacy policies. Binary assessments in classes like valid or void are less desirable in the domain of privacy policies where, especially before the introduction of the GDPR, much of what was legally allowed was still undesired by users.

Taxonomy
For a topic classification approach, i.e., a supervised approach, rather than an unsupervised topic modelling approach, a taxonomy of clause topics is needed. At first, this might seem like a limitation of the approach, because, in theory, a contract can regulate arbitrary aspects. In practice, however, standard form contracts underly strong lim-itations, because, under EU jurisdiction, clauses that are "unexpected" are automatically void under the Unfair Contract Terms Directive (93/13/EEC). Therefore, if a taxonomy is extensive enough, the information that a clause is not covered by one of the topics in the taxonomy is already important information in itself, because it means that the clause is very likely void.
To build such an extensive taxonomy for T&C from online shops, we used contract templates from legal literature (Sommer and von Stumm, 2017;Fingerhut, 2009) and industry associations (IHK Munich and Upper Bavaria, 2020;Schirmbacher, 2018), as well as a commercial T&C generator 1 and analysed which topics are present in these templates, because they are used by many online shops.
For each of these sources, two legal experts with experience in consumer protection law went through the templates and annotated each clause with one or more fitting topic and zero or more fitting subtopic label(s). In the end, the topic labels from the different annotators were aligned by the authors. By the combination of the abovedescribed sources, we derived a taxonomy of 22 classes (topics) and 36 sub-classes (subtopics). The differentiation between topics and subtopics was mainly based on the structure of the sources, i.e. the organisation in sections and subsections in the templates. A topic, for example, could be "delivery" and subtopics could be the delivery time or the delivery costs.
To get an estimate of how extensive our taxonomy is, we used it to manually annotate more than 6.000 clauses from real T&C (see section 4). Of these more than 6,000 clauses, the taxonomy was able to cover 90.08%. The remaining clauses (336) fell into only two classes: 285 clauses contained information regarding vouchers and gift cards, and 51 clauses contained information about codes of conduct. We added both classes to the taxonomy. The final taxonomy, therefore, consists of 23 labels for topics and 37 labels for subtopics. All labels in the taxonomy are shown in Table 2. "Geizhals" 3 ) that also offer a localised version of their respective websites in English, targeted to the British market 4 . On these websites, shop operators manually report the URLs to their T&C, which we extracted with a web-crawler. We randomly selected 142 German T&C and 30 English T&C from these pages. Each clause from these contracts was subsequently copied into an Excel file, in which each row contains one clause. In addition to the text of the clause itself, each row contains a unique id, an id for the contract the clause belongs to, (if existing) the title of the superordinate paragraph and (if existing) the title of the clause.

Size
The corpus we built consists of 5,020 German clauses and 1,040 English clauses. In both languages, a contract, therefore, consists of roughly 35 clauses on average. All German clauses together consist of 351,903 words, which is an average of 2,478 words per contract (see Table 1). The English corpus contains 55,392 words which equals to an average of 1,846 words per contract. This means German clauses are, on average, significantly longer than English ones. 5,013 clauses (or 99.9% of all clauses) in the German corpus have a paragraph or clause title (or both), which we can use for the topic classification. In the English corpus, that is the case for 989 clauses or 95.1%.

Annotation
Each clause of both corpora was labelled with its topics and subtopics according to the taxonomy described in Section 3. First, each clause was labelled by a student using only the classes from the first level (topics) of the taxonomy. Then, where applicable, classes from the second level of the taxonomy (subtopics) were added. In a second step, this process was repeated by the authors, i.e., each clause was again first labelled with classes from the first level and then, where applicable, with classes from the second level. A clause can be labelled with more than one topic and subtopic. A clause can also be assigned to a topic without necessarily having to be assigned to a subtopic of it (but not the other way round). An example of such a clause from the corpus is "The warranty is subject to the relevant statutory provisions.", which is assigned the topic warranty, but not to one of its subtopics. 3 www.geizhals.de 4 www.idealo.co.uk, www.skinflint.co.uk Cases where the two annotators disagreed, were presented to (and finally decided by) consumer protection lawyers with many years of experience in advising consumers. The inter-annotator agreement was relatively high at 87%, i.e., only 13% of all clauses had to be decided by the lawyers. In this way, four people together spend more than 100 hours and generated more than 24,000 labels, which were consolidated into two corpora, one for each language, with 11,777 labels in total. The distribution of topics and subtopics is shown in Table  2.
The annotation also revealed local differences, e.g., almost none of the English contracts contained a model withdrawal form, the only two that did contain such a form were from companies based in Germany, while many German contracts contained one. On the other hand, clauses about loyalty schemes were almost non-existing in German contracts and far more popular in the English corpus. It is worth reminding that our English data set was collected specifically from a UK perspective, i.e., the shops are either based in the UK or specifically targeted at the UK market. English contracts from other markets, like the USA or Australia, would most likely look very different. Since the T&C we annotated are protected by copyright law, we are, unfortunately, not able to publish the corpus.

Approaches
We compared seven different approaches to the classification of clause topics in standard form contracts from German and English online shops: Rulebased keyword matching, Logistic Regression, Random Forest, Multilayer Perceptron (MLP), Long short-term memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT). In the following sections, we will shortly introduce how we used the different approaches. For each of the approaches, we trained two classifiers, one which only classifies topics and one which only classifies the subtopics. Experiments we conducted with joint classification models, i.e., models that classify topics and subtopics at the same time turned out to decrease the classification quality for both, topics and subtopics.
We split both corpora into a training (80%) and a test (20%) set, using scikit-multilearn (Szymański and Kajdanowicz, 2017) to make sure the representation of labels is balanced between the training and the test set and reflects the original distribution.  For the stochastic approaches, we performed a grid search with a k-fold cross-validation on the training data to find the optimal parameters for each approach.
Our initial hypothesis, based on similar research, was, that with increasing complexity of the models, the performance would also increase, i.e. we expected BERT to perform best, followed by LSTM, MLP, and the "classic" ML approaches.

Rule-based
As a baseline, we first developed a rule-based classification approach. We used a simple keywordmatching approach. For each topic and subtopic in the taxonomy, we asked the consumer protection lawyers to provide a list of keywords that are distinctive for the topic/subtopic. The list can contain independent keywords (OR), keywords that should appear together (AND), and keywords that should not appear (together) (NOT).
We pre-processed the clauses using SoMaJo (Proisl and Uhrig, 2016) to split the clauses into sentences and the sentences into tokens. Afterwards, we lemmatised all tokens using the Stanford Lemmatizer  for English and the Mate tools Lemmatizer (Björkelund et al., 2010) for German before applying the rules. In German, we noticed that lemmatisation (but also stemming) face big challenges, especially in the legal domain, when it comes to compound nouns, i.e., nouns that are combined to create new nouns, like "Vertragspartner" (contractual partner) is a combination of "Vertrag" (contract) and "Partner" (partner). Compound nouns can be inflected internally ("Vertragspartner"), and splitting them into their constituents is not trivial. A "Druckerzeugnis" (printed matter) could, for example, lexically speaking either be a "Druck-Erzeugnis" (print -matter) or a "Drucker-Zeugnis" (printer -certificate). While there are existing approaches on how to automatically split compound words into their respective parts (e.g., by Baroni et al. (2002), Koehn and Knight (2003), Daiber et al. (2015), Sugisaki and Tuggener (2018), and Weller-Di Marco (2017)), the problem is far from being trivial and is not yet addressed in our implementation.

Logistic Regression
Second, we trained a logistic regression classifier, which we implemented using Scikit-learn (Pedregosa et al., 2011). As input, we used a Tf-idf vector representation of the concatenation of clause text and titles. Before transforming the clauses into these vectors, we removed stopwords using "Stopwords ISO" 5 . Since logistic regression does not inherently support multi-label classification, we used a "one-vs-the-rest" approach. Instead of training one classifier, we train one classifier for each class, which performs a binary classification against all remaining classes and combined all results to decide which labels are predicted for a given input.
We grid search with a 10-fold cross-validation on the training data to find the best parameter for the regularisation strength for the classification of topics. We performed multiple iterations on both languages to narrow down the search space. In German, we achieved the best results with C = 1, 000 and in English with C = 45, 000. The values are rather high for both languages but especially for the smaller English data-set. Since C is the inverse of the regulator (1/λ), a high value for C means a low value for λ and hence poses the risk of overfitting. We performed the same procedure for the classification of subtopics and found that C = 100 performed best in both languages, which is significantly lower and therefore less prone to overfitting.

Random Forest
Logistic regression is computationally efficient and generalises well, and is, therefore, a good baseline. However, its inability for "real" multi-label classification is a drawback in our use case. Decision trees do inherently support multi-label classification and also are inherently explainable. However, they are not as efficient as logistic regression and are more prone to overfitting. Instead of training just one decision tree, we use a random forest approach, where multiple independent randomised  decision trees are trained, and a majority vote is used for classification. As input, we again used Tf-idf vectors. We again performed a grid search with stratified 10-fold cross-validation on the training data to find the best performing values for the parameters: number of estimators (i.e., the numbers of trees), the maximum depth of the trees, the minimum number of samples per internal node that is needed for a split, and the minimum number of samples per leaf. As usual, we performed several iterations to narrow down the search space before, in the final iteration, we found the following values to perform best. In German: number of estimators = 2,000, maximum depth = ∞, samples per node = 2, samples per leaf = 1 and in English: number of estimators = 1,000, maximum depth = 100, samples per node = 2, samples per leaf = 1.

Neural Networks
For the different approaches using neural networks, we also evaluated different input encodings, namely different kinds of word embeddings. To train domain-specific embeddings, we used a larger corpus than the one described in Section 4, because the data does not have to be annotated. We collected the corpus in the same way as the other corpus from the price comparison websites, however, it is more than 30-times bigger, consisting of 5,412 contracts, 4,869 in German and 543 in English.
We used the following embeddings: • German -Word2Vec embeddings with 300 dimensions based on the German Wikipedia 6 -GloVe embeddings with 300 dimensions based on the German Wikipedia 7 -Word2Vec embeddings with 300 dimensions we trained from scratch on the above described corpus of T&C

• English
-Word2Vec embeddings with 300 dimensions based on the Google News Corpus (Mikolov et al., 2013) -GloVe embeddings with 300 dimensions based on Wikipedia and Gigawords 5 (Pennington et al., 2014) -Word2Vec embeddings with 300 dimensions we trained from scratch on the above described corpus of T&C Training neural networks is computationally much more expensive than, e.g., logistic regression and at the same time depends on more parameters. To manage the increasing complexity, we changed our approach for the parameter optimisation by reducing the 10-fold cross-validation to a 5-fold cross-validation. Additionally, we fixed parameters that always performed best or almost best, independent from other parameters, as early as possible in order to reduce the search space and converge faster to a local optimum.

MLP
The hyper-parameters we optimised of the MLP were the number of layers, the number of neurons per layer, the dropout, the batch size and the number of epochs. The results of the hyper-parameter studies can be found in Appendix A.

LSTM
For the LSTM network we optimised the sequence length, the number of LSTM layers and the number of neurons in them, the number of dense layers and the number of neurons in them, the dropout, the batch size, and the number of epochs. The best performing parameters we found for the topic and subtopic classification can be found in Appendix A.

BERT
Finally, we evaluate an approach using a transformer model for the clause topic classification, more specifically, the BERT language model (Devlin et al., 2019). We used the HuggingFace transformers library (Wolf et al., 2019) to fine-tune the pre-trained language models and implement the classification.
For English, we used the "bert-base-uncased" pre-trained model, provided by the original authors Devlin et al. (2019). The model, which is trained on lower case English texts, has 12 hidden layers with a size of 768, 12 attention heads per attention layer, and 110 million parameters. For German, we used the "bert-base-german-cased" model from Chan et al. (2020). It is trained on cased German texts and, like the original model, has 12 hidden layers with a size of 768, 12 attention heads per attention layer, and 110 million parameters.
The original BERT language model was trained on the English Wikipedia and the BookCorpus by Zhu et al. (2015), which consists of 11,038 fiction books that are available for free on the internet. The German language model we are using was pretrained on a more diverse set of sources, among which are the German Wikipedia and a web corpus gathered by Suárez et al. (2019), which account for more than 90% of the data the model was trained on. However, the model was also trained on the Open Legal Data set from Ostendorff et al. (2020), which consists of more than 100,000 German court decisions. We also briefly evaluated a multilingual approach with the Multilingual Universal Sentence Encoder transformer model, which was used by Braun and Matthes (2020) for the multilingual automated detection of T&C, however, first tests on German and English were not promising, so we did not follow through on the approach.
We used our training data to fine-tune both language models, the English and the German, for the topic classification task. In order to find the best hyper-parameters, we split 20% off the training data as validation set. We started our search with the values suggested in the original BERT paper: batch size 16 or 32, learning rate 5e-5, 3e-5 or 2e-5, and 2, 3 or 4 epochs (Devlin et al., 2019). However, the authors also note that the optimal hyper-parameters are task-specific and that small data sets (which they define as less than 100,000 labels) are more sensitive to the choice of parameters than larger ones. For our data sets and task, we found a smaller batch size with a slightly higher number of epochs to work better than the suggested parameters in both languages. We found a batch size of eight and a learning rate of 5e-5 to perform best for the topic and subtopic classification in both languages. In German, eight epochs performed best for the topic classification and six for the subtopic classification. In English, six epochs for the topic classification and 21 epochs for the subtopic classification performed best. All other parameters were kept equal to the original pre-trained model.

Evaluation
Our baseline approach of using keywords achieved an F1-score of 0.78 in German for topic classification and 0.64 for subtopic classification. The performance in English was worse with an F1score of 0.72 for topics and 0.46 for subtopics. We noticed that, in German, the list of keywords   mostly consisted of domain-specific compound nouns, like "Widerrufsrecht" (right of withdrawal) or "Gefahrenübergang" (transfer of risk), which are very distinctive for their respective topics and make the classification relatively easy. This is also a possible explanation for why the logistic regression classifier, with Tf-idf vectors as input, performed so well on the German corpus. With an F1-score of 0.87 for topics and 0.84 for subtopics, it was only surpassed by the BERT model. All other approaches performed comparable to each other in German, with F1-scores between 0.8 and 0.83 for topic classification and 0.72 to 0.77 for the subtopic classification (see Table 3 and 4).
In English, the picture was less clear, with logistic regression still performing best for topic classification (F1-score 0.80) but being surpassed by the neural network approaches for subtopic classification. However, the clear overall winner, with the best performance on both languages and classification levels was BERT, which scored up to 0.91.
For the approaches we evaluated with different inputs, i.e. the MLP and LSTM, the values in Table  3 and 4 represent the best results achieved. There was no clear pattern visible of which word embedding model performs best. All of them achieved comparable results and no one performed best or worse in all settings.

Transferability
Due to their practical relevance and availability, we focused on T&C from online shops in this paper. However, there is no reason why the same technology could not be applied to other types of standard form contracts, e.g. from banks and insurances. At the same time, the taxonomy we developed and the models we trained have a component that is specific to online shopping, broken packaging, for example, is a topic, that is not relevant for banking.
To get an idea of how domain-specific the taxonomy and the models are, we annotated the T&C of three of the largest German banks (Commerzbank, Deutsche Bank, and Sparkassen) in the same way described in Section 4.2: first, a student, then authors annotated the all clauses of the contracts independently with their topics and subtopics, then conflicting labels were resolved by the team of experts. We used the taxonomy described in Section 3 for the classification, however, we added a class "n.a." to mark clauses that cover a topic that is not represented by any of the classes in the taxonomy.
The three contracts consist of 214 clauses, about 71 clauses per contract, and 13,681 words, an aver-  The annotation process confirmed this assumption. Of the 214 clauses, only 71 (or 33%) are concerned with a topic that is covered by our taxonomy (see Table 5). The other clauses are concerned with a wide range of banking specific topics, from deposit protection funds to banking confidentiality. This means that, even if we would correctly classify all the other clauses, we could never achieve a recall above 0.33. We can already conclude, that the taxonomy we developed can not simply be applied to other types of standard form contracts without adaption.
Since the taxonomy is still able to cover onethird of the banking contracts, we wanted to test how well the classifiers we trained would perform on this data set. Therefore, we took the best performing topic classifier, i.e., the BERT model, and applied it to the new corpus. The evaluation of the results is shown in Table 6. We can see that for some of the topics, e.g., applicability, applica-bleLaw, arbitration, and changes, the performance is very good, even tough our model has never seen this type of contract before.

Conclusion
In this paper, we compared different approaches to classify the topics of clauses in standard form contracts from online shops, based on a taxonomy  of clause topics and subtopics we developed and a bilingual corpus of more than 6,000 clauses we gathered and annotated. Our evaluation showed, that our initial hypothesis, that the model performance would increase with complexity, did not hold.
While BERT did indeed perform best for both languages, the much simpler logistic regression approach showed the second-best performance. Considering the computational time and power that is needed to not only train the more complex model but also during inference of the labels, the simple logistic regression approach might for some practical application be the better choice.
We were surprised to find that multilingual approaches, using the German and English data together, did not seem to bring any improvements for this task, even though earlier work in the legal domain, e.g. by Niklaus et al. (2021) and Braun and Matthes (2020), have shown that multilingual models can improve performance. This aspect needs further investigation.
While the taxonomy we developed and the models we trained are domain-specific for eCommerce, first tests suggest that the approaches can be transferred to other types of standard form contracts and that event the models can partially be transferred to other domains, at least for more "technical" clauses concerning the contract itself. This could apply to all types of consumer standard form contract within a highly regulated domain, like insurances, housing, and employment, and is something we would like to investigate further in the future.