{'fraction_non_alphanumeric': 0.02648239654107474, 'fraction_numerical': 0.01852995676343422, 'mean_word_length': 4.521099744245524, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DoSSIER@COLIEE 2021: Leveraging dense retrieval and summarization-based re-ranking for case law retrieval

In this paper, we present our approaches for the case law retrieval and the legal case entailment task in the Competition on Legal Information Extraction/Entailment (COLIEE) 2021. As first stage retrieval methods combined with neural re-ranking methods using contextualized language models like BERT achieved great performance improvements for information retrieval in the web and news domain, we evaluate these methods for the legal domain. A distinct characteristic of legal case retrieval is that the query case and case description in the corpus tend to be long documents and therefore exceed the input length of BERT. We address this challenge by combining lexical and dense retrieval methods on the paragraph-level of the cases for the first stage retrieval. Here we demonstrate that the retrieval on the paragraph-level outperforms the retrieval on the document-level. Furthermore the experiments suggest that dense retrieval methods outperform lexical retrieval. For re-ranking we address the problem of long documents by summarizing the cases and fine-tuning a BERT-based re-ranker with the summaries. Overall, our best results were obtained with a combination of BM25 and dense passage retrieval using domain-specific embeddings.


INTRODUCTION
In case law systems, precedent cases are a primary legal resource for the decision of a new given case. Therefore it is a part of a lawyer's or paralegal's daily work to find precedent cases which support or contradict a new case [24]. With the exponentially growing amount of electronically stored information in the legal domain [25], it costs legal professionals increasingly more effort to retrieve the cases which are relevant to their case. In order to find evidence lawyers require their search systems to find all cases which are relevant [1], but at the same time, legal researchers will examine up to 50 results in practice [19] and therefore require a precision-oriented solution. We tackle these requirements of legal case retrieval in Task 1 of the Competition on Legal Information Extraction/Entailment (COL-IEE) 2021 by first retrieving candidates from the whole corpus with the aim of a high recall and then re-ranking these candidates for precision at high ranks. In Task 2 of COLIEE 2021 it is the task to identify a paragraph of an existing case that entails the decision of a new case [14] and we approach this problem with re-ranking as well.
In news and web search, contextualized language models like BERT [5] brought substantial effectiveness gains to the first stage retrieval [6, 8-10, 16, 27] as well as to the re-ranking stage [2,7,12,13]. We aim to transfer these advantages also to the task of case law retrieval, however this task has specific challenges compared to retrieval tasks in the web and news domains: documents are written in domain specific language [24], there is a specific notion of relevance not only on document-but also on paragraph-level [25] and the documents tend to be long [21,25]. For example in the case law corpus from COLIEE 2021 the documents contain on average 1274.62 words,with minimum 1 word and maximum 76, 818 words. This is not only the case for the candidate documents in the retrieval collection, but also for the query cases. As the input length of BERT is limited, we propose two different approaches for handling the longer documents with BERT, one for first stage retrieval and one for re-ranking. In order to achieve a high recall for the first stage retrieval, we reason that prior cases are not only relevant on a document-level to a query case, but that a prior case can be relevant to a query case only based on a single paragraph which is relevant to another paragraph of the query case [20,21,26]. Therefore we propose for the first stage retrieval to split up the query case and cases in the corpus into their paragraphs and retrieve for each paragraph of the query case relevant prior cases based on the relevance of their paragraphs. For the document-level and paragraph-level retrieval, we evaluate both lexical retrieval model and of a semantic retrieval model. For the lexical retrieval model we choose BM25 [17] and for the dense passage retrieval model we choose DPR [9]. We train the dense passage retrieval model on the legal entailing paragraph pairs as we suggest that the relevance of the paragraphs to each other is crucial for the relevance on the document-level. We denote the trained DPR model with lawDPR. When comparing the paragraph-level and documentlevel retrieval, we demonstrate that the paragraph-level retrieval achieves a higher recall for BM25 as well as for lawDPR and that lawDPR outperforms BM25 in terms of retrieval recall. We also show that the results of the lexical and semantic retrieval of BM25 and lawDPR complement each other and that the score combination of both retrieval models leads to superior results. For re-ranking the retrieved results we experiment with the approach of summarizing the query cases and the cases in the corpus as Rossi and Kanoulas [18], Tran et al. [23] have shown that query document summarization is valuable for case law retrieval. For this, we fine-tune Longformer Encoder-Decoder (LED) as a stateof-the-art abstractive summarization model [3] and use this for summarizing the cases. For each query and document case, we summarize the text and interpret the re-ranking as a binary classification problem and fine-tune BERT on predicting whether the summarized query case is relevant to a summarized case in the corpus or not. For Task 1 we submit 3 runs: one based on the ranking with BM25, one with the combination of BM25 and lawDPR and one with the first stage retrieval of BM25 and lawDPR and with re-ranking with BERT. For the legal entailment task (Task 2) we evaluate BM25 [17] and the DPR model [9] trained on the entailing paragraph pairs ('lawDPR').
Here we find that the combination of the BM25 and DPR scores also improves the overall performance for identifying the entailing paragraph. For Task 2 we submit 3 runs: one based on the ranking on BM25, one based on the ranking of lawDPR and one with the combination of BM25 and lawDPR. We make the source code available at: https://github.com/sophiaalthammer/dossier_coliee.

TASK DESCRIPTION 2.1 Task 1: The Legal Case Retrieval Task
The aim of legal case retrieval is to design systems to automatically identify the supporting cases of a given query case, which should be noticed for solving the query case [14]. The task consists of reading a new case and selecting supporting cases 1 , 2 , . . . , ("noticed cases") from the whole case law collection for the decision of .

Task 2: The Legal Case Entailment Task
In the legal case entailment task, the goal is to design a system that finds paragraphs in a relevant case that entail the decision of a given new case [14]. In the Task 2 of COLIEE there is a query paragraph given as well as the paragraphs of one legal case as candidates and the task is to find the paragraphs from the legal case which entail the decision of the query paragraph.

Training and test collection
The training and test collections for Task 1 and Task 2 contain cases from the Federal Court of Canada case laws. For Task 1, a corpus with 4415 legal cases is given from which relevant cases should be retrieved for each query case. The cases can contain as addition to the English version also a French version. For Task 2 there are for each query paragraph the paragraphs of one case as candidates given and one needs to identify the entailing paragraphs. The statistics of the collections are in Table 1 including the number of queries, the average number of relevant documents and the average length of the queries and candidates.

Task 1
3.1.1 First stage retrieval. For the lexical retrieval in the first stage retrieval we use BM25 [17]. When querying the index with query , the BM25 model assigns each document in the index a ranking score 25 ( , ), the higher the score, the higher the relevance of the query and the indexed document. For semantic retrieval, we use a dense passage retrieval model [9] based on two Siamese BERT Encoders, one encodes the query passage, the other one the candidate passage. The encoder encodes the query and candidate passage into a vector and the dot-product between those two vectors denotes the relevance score ( , ) between the query and the candidate passage . This dense retrieval model (lawDPR) is trained on the entailing query-paragraph pairs of Task 2, in order to align the vector representations of entailing paragraphs. As the maximum input length for the dense passage retrieval model [9] is limited, we employ this model on the full text of the case by splitting up the whole document into its paragraphs 1 , .., and indexing each paragraph separately. When querying the paragraphlevel index, we also split up the query case into its paragraphs 1 , .., and retrieve for each query paragraph with ∈ [ ] the ranked list of paragraphs 1 , .., . The paragraph in the ranked lists with ∈ [ ] are reduced to the paragraphs documents and then the ranked lists of documents are aggregated to one ranked list of documents for the whole query case. This aggregation will be further investigated in the next section. Equivalently we also use this approach for retrieval with BM25 on the paragraph-level and refer to this approach to paragraph-level retrieval.
In order to analyze the impact of the paragraph-level retrieval compared to document-level retrieval on the retrieval effectiveness, we also conduct document-level retrieval with BM25 and lawDPR.
Here all documents in the corpus are indexed based on their whole text and the query case is also encoded with all its text. As the dense passage retrieval model has a limited input length and therefore can only retrieve passages up to 512 tokens, the document-level retrieval for lawDPR is the retrieval based on the first 512 tokens of the query case and the candidate case.
Aggregation. For the retrieval on the paragraph-level we retrieve for each paragraph of the query document 1 , .., a ranked list 1 , .., of top results. These ranked lists with ∈ [ ] of paragraphs must be aggregated to one overall ranked list for the whole document, this process of aggregation is visualized in Figure  1. For the aggregation we reduce the paragraphs per ranked list to the paragraphs' document. As the paragraphs are reduced to their documents it is possible that one document occurs multiple times in the list. Within this reduction from the paragraph to its document the scores of the retrieval model ( , ) of each paragraph in the ranked list are replaced with an integer scores of the document ( , ) according to their position: The first document gets the score , per position in the list the score decreases by 1 so that the last document gets the score 1. Then we aggregate the lists per query paragraph to one ranked list for the whole query document by adding the integer scores for each retrieved document and ranking by the new additive scores. The new additive score of a document for a query is then: We also experiment with other aggregation strategies using the similarity scores of the retrieval model or using an interleaving approach, but find that the aggregation with the additive integer scores ( , ) for a document and query lead to the overall best performance.
Combination. We find that 67% of the relevant cases of Top1000 which are retrieved from lawDPR on paragraph-level and 70% of the relevant cases of Top1000 which are retrieved from BM25 on paragraph-level are the same documents. Therefore 23% of the retrieved relevant cases from lawDPR and 30% of the retrieved relevant cases from BM25 are different, hence BM25 and lawDPR retrieve different relevant documents. For the first stage retrieval we combine for each document in the index the relevance scores of BM25 25 ( , ) and the relevance score for the whole document of lawDPR ( , ) from paragraph-level retrieval with BM25 and lawDPR. In order to get an overall score of a document and a query we add the BM25 and the lawDPR relevance scores with a scalar weighting , ∈ R We denote the combination with BM25+lawDPR.

Re-ranking.
Summarizer: LED. The current state of the art in abstractive summarization is Transformer models [11,15]. However, the input of pre-trained available models of these architectures is limited to 1024 tokens, and the majority of case law documents in our collection is longer than that. Beltagy et al. [3] proposed Longformer-Encoder-Decoder (LED), which is a Transformer variant that supports much longer inputs. For this competition, we evaluate the effectiveness of LED for case law retrieval.
BERT. For re-ranking we use BERT and fine-tune the pre-trained BERT model (BERT-Base, uncased) with a linear combination layer stacked atop the classifier [CLS] token on binary classification if a query summmary and a document summary are relevant to each other or not. We represent the query as sentence A and the document as sentence B in the BERT input: Cut-off value. As the task is to retrieve the relevant cases to a given query , we consider the top-k-ranked documents in the ranked list as relevant and denote as cut-off value. We evaluate the best cut-off value depending on the F1-score of the validation set.

Task 2
For identifying the entailing paragraphs to a given query paragraph we use lexical and semantic ranking approaches in order to rank the given candidate paragraphs . In order to predict which paragraphs of the ranking are entailing the query paragraph, we consider the top-k-ranked paragraphs, where we denote as cutoff value. We evaluate the best cut-off value depending on the F1-score on the validation set. In order to rank each paragraph given the query paragraph, we create per query paragraph an index containing the given candidate paragraphs. Then we query this index with the query paragraph and obtain the ranking for the candidate paragraphs. We do this procedure either using BM25 or lawDPR in order to compare both models for the legal entailment task.
lawDPR. For the semantic retrieval we use the same dense retrieval model [9] as in Task 1, which is trained on the entailing query-paragraph pairs of the training collection for Task 2. As Task 2 is on paragraph-level we can directly use the dense passage retrieval model without aggregating the results as in Task 1.
BM25+lawDPR. We also combine the ranking of BM25 and lawDPR with the same method as for Task 1.

EXPERIMENT DESIGN 4.1 Data pre-processing
For the experiments we divide the training collections of Task 1 and Task 2 into a training and validation set. The validation sets for Task 1 and for Task 2 contain the last 100 query cases of the training sets, respectively. For the data pre-processing for Task 1, we remove the French versions of the cases when reading in the cases. We divide the cases into an introductory part, a summary, if they contain one, and their paragraphs. As we want to distinguish the cases by their text, we remove text parts of the cases which appear exactly the same in multiple cases, as they do not add any information which is particular for one case. Therefore we remove introductory parts and summaries, which appear exactly the same in more than 100 cases from the case text as we consider them as non-informative. The introductory parts have an average length of 73.66 words, the summaries have on average 227.05 words and the paragraphs have a average length of 92.80 words. We use the same data pre-processing for all submitted runs. We pre-process the data of Task 2 by reading in the paragraphs, splitting the words at whitespaces and removing tabs. Here we also use the same data pre-processing for all submitted runs.

First stage retrieval.
BM25. We use the BM25 implementation from ElasticSearch 1 with default parameter values = 1.2 and = 0.75. lawDPR. We train the dense retrieval model with two BERTbased-uncased Siamese Encoders on the training set and validation set of Task 2 in the same fashion as in [9]. Here a pair of query and candidate paragraph which are relevant to each other are considered as positive sample and a pair of query and candidate paragraph which are not relevant to each other are considered as negative sample. Different to Karpukhin et al. [9] we sample the negative candidate paragraphs to a query paragraph randomly from the paragraphs which are not denoted as relevant to this query paragraph. We train the model for 40 epochs with a pairwise loss for the first 30 epochs and a list-based loss for the last 10 epochs, in order to include the pairwise and listwise relation of the samples in the training. We use batch size of 22, a maximum sequence length of 256 and a learning rate of 2 * 10 −5 .
We index the corpus on the paragraph-and on the documentlevel using BM25 and lawDPR and then query each index with the query cases. For the paragraph-level retrieval the query is split up into its paragraphs, for each query paragraph a ranked list of paragraphs is retrieved and aggregated to a ranked list of documents as described in section 3. For the first stage retrieval on the paragraph-level the introductory part and the summary are also treated as paragraph of the document. For the document-level retrieval the documents are indexed based on their whole text and for each query case the whole text is taken into account.
BM25+lawDPR. We experiment with combining the scores of BM25 and lawDPR as described in section 3. We take the Top1000 aggregated lists for the paragraph-level retrieval of BM25 and lawDPR and use the weights

Re-ranking.
Summarizer: LED. For LED, we used the Huggingface transformers library 2 and set the local attention window size to 512 tokens. To limit memory use, we use gradient checkpointing and set the input size in training to 8192 tokens which covers more than 86% of COLIEE'18 Task 1 documents completely (the longer documents are truncated at 8912 tokens). We set the maximum length to generate a summary for an unseen document as 10% of the length of the original text. We use the COLIEE'18 Task 1 data, which contains human reference summaries, for evaluation of the summarizers 3 .
In COLIEE'18, the summaries are provided for all queries and more than 80% of document cases. We extracted summaries like [22], and after removing duplicates, 6, 257 unique documents are left for which a summary is available. Finally, we fine-tuned the LED model on the unique documents that contains summary. We kept the other hyperparameters (optimizer, dropout, weight decay) identical to [3] and set the global attention on the first <s> token. We use the fine-tuned LED on COLIEE'18 to generate summary for query and   document cases in COLIEE'21, and provide the summaries as the representation of cases in BERT-based re-ranking step.
BERT-based re-ranking. We truncate the summarized documents such that the concatenated query document (truncated at 100 words), candidate document, and the separator tokens do not exceed 512 tokens. We re-rank the top-500 BM25+lawDPR results. We find = 7 as the optimal cut-off of the ranking for the selection of documents. We train each model for 100 epochs, each with 32 batches of 16 training pairs, with the initial learning rate of 3 * 10 −5 , followed by a power 3 polynomial decay.
For Task 1 we submit 3 runs: one based on the paragraph-level retrieval and ranking of BM25, the second based on the combination of paragraph-level retrieval of BM25 and lawDPR and the third takes the retrieved candidates from the combination of BM25 and lawDPR and re-ranks them using the summarizer and BERT.

Task 2
BM25. We use the BM25 implementation from ElasticSearch with default parameters and create an index for each query paragraph and its candidates.
lawDPR. We take the trained lawDPR model from Task 1.
BM25+lawDPR. We combine the scores of BM25 and lawDPR as described in scetion 3 and use the weights [ , ] ∈ { [1,1], [2,1], [3,1] , [4,1]}. For Task 2 we submit 3 runs: one based on BM25, the second based on lawDPR and the third with the combination of BM25 and lawDPR. We compare the first stage retrieval of BM25 and lawDPR on the paragraph-and the document-level and also evaluate the combination of the scores of BM25 and lawDPR. In the first stage retrieval it is our goal to achieve a high recall, therefore we evaluate the recall@100, recall@200, recall@300 and recall@500 using pytrec_eval 4 on the validation set. The results are shown in Table 2. As described in section 4 we combine the scores of BM25 and lawDPR and obtain a list ranked by the combination of the BM25 and lawDPR scores. When evaluating the retrieval on the validation set, we find that the combination with the weights [3,1] leads to the best results, therefore we only present the results for this weighting and denote it as BM25+lawDPR.
Comparing the retrieval on paragraph-and document-level we see that the retrieval on paragraph-level outperforms the retrieval on whole document-level for BM25 as well as for lawDPR. This shows that our approach of tackling the long documents for a contextualized language model with limited input length is not only beneficial for the dense passage retrieval model, but also for the lexical retrieval with BM25. Furthermore we see that BM25 outperforms the first stage retrieval on paragraph-level of lawDPR by 4 − 6% in terms of recall@N. However when combining the scores of BM25 and lawDPR, we see that the overall recall is the best at all evaluated cut-off values.

Ranking.
We evaluate the precision, recall and F1-scores for the runs we submitted at the cut-off value of 7 for each run as we find in out analysis in Figure 2 that the best cut-off value in terms of F1-score is 7 for all three runs. The evaluation results on the validation set are in Table 3 and on the test set in Table 4.
Here we see that the overall ranking performance is improved for  the validation set of BM25+lawDPR, this approach also outperforms the re-ranking based on the summaries. Contrary to that we find that on the test set BM25 achieves the best performance in terms of precision, recall and F1-score. There is a difference in the evaluation scores between our evaluation and the evaluation of the task coordinator. We use for our evaluation the off-the-shelf pytrec_eval 5 library and were in contact with the task organizer, however we could not clarify the differences between the evaluations.

Task 2
For Task 2 we evaluated precision, recall and F1-score on the evaluation set for multiple cut-off values. The overall F1-score for multiple cut-off values is visualized in Figure 3, the performance for the precision and recall of various cut-off values is visualized in Figure  4. Figure 4 shows how the precision decreases with an increasing cut-off value and therefore an increasing recall. Here we also see that BM25 and lawDPR have a similar performance, with lawDPR having a better performance with a higher cut-off value than BM25. We also clearly see the gap between BM25 and lawDPR and the combination of the scores of BM25 and lawDPR. We see a similar picture in Figure 3, here the F1-score is continuously decreasing with a higher cut-off value, therefore we select a cut-off value of 1 5 https://github.com/cvangysel/pytrec_eval  for each run. We also see that the overall performance is the best with the weight [4, 1] therefore we use these weights for combining BM25 and lawDPR. In Table 5 we can see the precision, recall and F1-score on the validation set for BM25, lawDPR and BM25+lawDPR. Here we see that BM25 has a better performance than lawDPR, but that both runs are outperformed by the combination of BM25 and lawDPR. This also suggests that combining the strengths of lexical and semantic retrieval models is beneficial for legal paragraph entailment and that BM25 and lawDPR complement each other. We also evaluate our three final runs on the test set, the evaluation results can be found in Table 6. Here we find the same relations between the evaluation results as for the validation set evaluation.

CONCLUSION AND FUTURE WORK
Our participation at COLIEE 2021 in Task 1 and Task 2 gave the opportunity to explore information retrieval challenges in the legal case law retrieval and legal entailment. Our goal is to combine traditional lexical retrieval models with dense passage retrieval models as well as use contextualized re-ranking models for re-ranking the results. For legal case retrieval we identify the challenge of long documents for dense retrieval and neural re-ranking strategies. Therefore we present, compare and evaluate two approaches for handling the long documents: • Splitting up the documents into their paragraphs and proposing a paragraph-level retrieval for first stage retrieval and • Generating summaries for the neural re-ranking with BERT.
We show that the paragraph-level retrieval in the first stage outperforms the document-level retrieval and that the combination of lexical and semantic retrieval models leads to the best results. Also for the ranking we find that the combination of lexical and semantic models improves the overall effectiveness of the ranking. Our experiments with re-ranking based on the summaries of the cases seems not to improve the overall ranking effectiveness. Furthermore we also find that the combination of lexical and semantic retrieval methods improves the overall performance also for the passage entailment task. This leads to the conclusion that lexical and semantic retrieval methods have different strengths and complement each other. In future we plan to investigate dense retrieval models with domain specific contextualized language models like LegalBERT [4] and with different paragraph aggregation approaches.