{'fraction_non_alphanumeric': 0.030580766307506595, 'fraction_numerical': 0.006125095006035678, 'mean_word_length': 4.926868044515103, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Legal Knowledge Extraction for Knowledge Graph Based Question-Answering

This paper presents the Open Knowledge Extraction (OKE) tools combined with natural language analysis of the sentence in order to enrich the semantic of the legal knowledge extracted from legal text. In particular the use case is on international private law with specific regard to the Rome I Regulation EC 593/2008, Rome II Regulation EC 864/2007, and Brussels I bis Regulation EU 1215/2012. A Knowledge Graph (KG) is built using OKE and Natural Language Processing (NLP) methods jointly with the main ontology design patterns defined for the legal domain (e.g., event, time, role, agent, right, obligations, jurisdiction). Using critical questions, underlined by legal experts in the domain, we have built a question answering tool capable to support the information retrieval and to answer to these queries. The system should help the legal expert to retrieve the relevant legal information connected with topics, concepts, entities, normative references in order to integrate his/her searching activities.


Introduction and Problem Statement
The legal ontology modelling method [4,2] is a relevant instrument for defining the legal concepts and relationships included in legal texts (e.g., hard law, judgment, soft law, etc.) but it is extremely expensive, it depends to the hermeneutic approach adopted by each scholar or community (e.g., common law vs. civil law), it is influenced by a strong localization due to the local jurisdiction (e.g., domestic regulation and local court action), by the cultural and social norms (e.g., concept of gender) and furthermore every time there is a new modification in the legal framework (e.g., new legislation) a refinement or (even worse) a whole extension of the ontology is required. On the other hand, the semantic web techniques are very useful in legal domain to detect relevant texts according to situations and concrete cases, or to filter the connected legislation among wide corpora. Legal ontologies are also important, in the legal rule modelling, for providing a common vocabulary of predicates and thus to permit interoperability between different legal knowledge engineers belonging to different institutions. Other important applications of legal ontologies are legal design and smart contract domains in order to refer to the same legal concepts and axioms inside of a community. For these reasons a hybrid solution is necessary in order to take benefits from the legal ontology information, but with a reasonable balance between information granularity and human effort. A light ontology, based on the language analysis of the text and on simple relationships between classes, is a very poor instrument for the legal domain, where the legal norms often include exceptions and odd situations (e.g., derogation, retroactivity, suspension). In fact, the core of a domain legal ontology is very detailed and accurate but it takes years for an accurate modeling and during this period too much modifications can change the legal scenario. In the meantime the Ontology Design Patterns (ODPs) [8,13] method could help to maintain a good methodological connection between light and foundational approaches [6] (bottom-up and top-down approaches). In our work we show an effective way to combine the two approaches by mapping to Ontology Design Pattern (ODP) a Knowledge Graph (KG) automatically extracted for performing Question Answering (QA). The results are provided in the form of a Knowledge Graph (KG) of templates aligned to wellknown legal ODPs, and structured in a way that would enhance the selection of relevant material for a specific case-law or legal situation. The aforementioned KG is extracted from regulations such as: Rome I Regulation EC 593/2008, Rome II Regulation EC 864/2007, and Brussels I bis Regulation EU 1215/2012. The alignment of the KG to the legal ODPs, plus the fact that the KG is extracted from regulations, makes the KG a sort of light ontology. This light ontology will be structured in a way that would be possible for the legal end-users (e.g., lawyer, judge, scholar, students) to easily query and explore the extracted information through a QA algorithm. In the future we intend to integrate our approach in existing tools 1 for legal document analysis, thus allowing to the legal experts to formulate relevant queries in order to orient the KG during the modelling phase and to correct/disambiguate some edges/nodes of the extracted KG.

International Private Law
The International Private Law (PIL) is a complex legal domain that presents frequent conflicting norms between the hierarchy of legal sources (e.g., national vs. European level), between legal domains (e.g., consumer law vs. labor law), between the procedures adopted (e.g., criminal law vs. civil law). After the Treaty of Amsterdam (1 May 1999), the legislative powers for judicial cooperation in civil and commercial matters were transferred to EU institutions with the aim to harmonize the following issues: i) which state court has jurisdiction in private matters having cross-border implications; ii) which domestic law is applicable in such matters, iii) and under which conditions can a foreign decision be recognized and enforced in another Member State. Scientific research on PIL reveals the need to create a bridge between European and national laws on this domain, accessing heterogeneous legal sources. The European project Interlex 2 intended to investigate this domain and to use technology to fill the gap between different legal sources. The need to fill such a gap between legal sources is so frequent in the PIL domain that the legal experts need to recall all the norms and to combine them using the theory of interpretation principles and the case-law based approach. For this reason, the classical databases and information systems based on full-text search or document classification or document clustering seem not to be effective. This is because the terminologies are different (e.g., "consumer" in consumer law, and "data subject" in data protection law), the normative references are consistent only inside the same domain, the mapping of concepts is difficult because they are not perfectly equivalent. For this reason our approach is to discover correlation between different terms and parts of the legal documents, and to use the ODP main classes for structuring a KG that can be queried by the experts.

Legal Ontology Design Patterns
In the legal domain, different researches [10,1] identified some basic ontology design patterns regularly used for modelling norms. i) Agent-role-time 3 ; ii) Event-time-placejurisdiction 4 ; iii) Agent-action-time [7]; iv) Object-document [12]; v) Legal deontic ontology [5] [10]. These patterns, combined with linguistic taxonomies, could provide a good solution for creating a bridge between the variants of the legal definitions and the conceptualization level [11].

Proposed Solution
We can extract Knowledge Graphs (KGs) from legal documents by exploiting the grammatical dependencies of their content, through an automated dependency parser. In order to make sense of the extracted information, making these KGs useful for exploration and Question Answering (QA), we should be able to guarantee some properties that would facilitate the interoperability of the KG with state-of-the-art deep-learning based QA algorithms. Considering that modern state-of-the-art QA algorithms have several limitations in terms of input and output size, the challenge is not trivial. Furthermore these QA algorithms are trained with natural language and not Resource Description Framework (RDF) triples.
Assuming that serialising natural language into RDF triples is a challenging openproblem, the simplest solution appears to be to abandon RDF serialisation in favour of natural language, but natural language is not as structured as RDF and performing QA over large natural language corpora is too expensive. This is why our proposed solution consists in ad-hoc KG extraction of triples in the form of textual templates rather than classical RDF, in order to preserve the natural language while structuring it into a proper graph aligned to external resources such as WordNet or ODPs. We think that effective abstract querying can be possible by structuring the KG as an ontology, giving it a solid backbone in the form of a taxonomy. In fact, being able to identify the type/class of a concept would allow to perform queries with a reasonable level of abstraction, making possible to refer to all the sub-types (or to some super-types) of a concept without explicitly mentioning them.
Our proposed solution, for extracting and making sense of complex information stored into natural language documents, is defined by the following steps: [i)] KG ex-traction, Taxonomy Construction, Ontology Design Pattern Alignment, KG question answering. 5

KG extraction
KG extraction is the extraction of concepts and their relations, from a natural language text, in the form of a graph where concepts are nodes and relations are edges. As mentioned before, we are looking for a way to extract KGs that somehow preserve the original natural language, preferring them over classical RDF graphs. This way we can easily make them inter-operate with deep-learning based QA algorithms and language models.
More in detail, we perform KG extraction by: 1. Analysing the grammatical dependencies of tokens extracted by Spacy's Dependency Parser, therefore identifying noun syntagms (concepts): the possible objects and subjects of the triples to extract.
2. Using the dependency tree to extract all the tokens connecting two different target concepts in a sentence, thus building a template composed by these connecting tokens (the order of the tokens is preserved) together with the target concepts (replaced with the placeholders "{subj}" and "{obj}", in accordance with their grammatical dependencies).
3. Creating a graph of triples where target concepts are subjects/objects and templates are predicates.
The resulting triples are a sort of function, where the predicate is the body and the object and the subject are the parameters. Obtaining a natural language representation of these template-triples is straightforward by design, by replacing the instances of the parameters in the body. An example of template-triple (in the form subject, predicate, object) is: "the applicable law", "Surprisingly {subj} is considered to be clearly more related to {obj} rather than to something else.", "that Member State". Furthermore, to increase the interoperability of the extracted KG with external resources, we performed the following extra steps: i) We assigned a URI and a RDFS label to every node of the graph. The URI is obtained by lemmatising the label. ii) We added special triples to keep track of the snippets of text (a.k.a. the sources) from which the concepts and the relations are extracted. iii) We added sub-class relations between composite concepts (syntagms) and the simplest concepts (if any) composing the syntagm. Because of the adopted extraction procedure, the resulting KG is not perfect, thus it may contain some mistakes caused by wrong grammatical dependencies or other issues. But, due to the fact that the original natural language is practically preserved, one would expect that such imperfection would not significantly impact on QA if the adopted neural networks are robust enough (e.g. being trained on very large corpora of real text).

Taxonomy Construction
In order to efficiently use, query and explore the extracted KG, we need to structure it in a proper way. We believe that effective abstract querying can be possible by structuring the KG as a light ontology, giving it a solid backbone in the form of a taxonomy. In fact, being able to identify the types/classes of a concept would allow to perform queries with a reasonable level of abstraction, making possible to refer to all the sub-types (or to some super-types) of a concept without explicitly mentioning them. The taxonomy construction phase consists in building one or more taxonomies, through Formal Concept Analysis (FCA).
In order to build a taxonomy via FCA, one simple approach consists in exploiting (as FCA's properties) the hypernyms relations of the concepts in the KG. We found that a naive way to extract such relations is through the alignment of the extracted KG to WordNet 6 , via a Word-Sense Disambiguation algorithm. Applying FCA, to the hypernyms of the aligned Wordnet concepts, produces a forest of taxonomies. Every taxonomy in this forest is a cluster of concepts rooted into very abstract concepts that we can use as label/identifier for the respective taxonomies.
The results we obtained for the three EU's regulations are quite interesting. In fact FCA is able to identify very few concepts clusters (taxonomies), and these clusters resemble the same core concepts our domain experts previously (and independently) identified for the regulation under study: person, claim and contract. More in detail, the main clusters obtained through FCA are about: • Legal Documents: a document that states some contractual relationship or grants some right. • Acts: something that people do or cause to happen. • Organizations: a group of people who work together. • Causal Agents: any entity that produces an effect or is responsible for events or results. • States: the way something is with respect to its main attributes.

Legal Ontology Design Pattern Alignment
With rich enough taxonomies we can improve the quality of the KG structure by aligning it to known legal Ontology Design Patterns (ODPs). We can perform this alignment easily, by manually mapping the roots of every taxonomy obtained via FCA (see Section 3.2) to relevant concepts of the design patterns we identified in section 2.2. This is feasible because the number of relevant concepts in the ODPs is very small (in the order of 10). The KG extraction is said to be a bottom-up approach (from concrete documents, to abstract ontologies), while the design of ontologies through patterns is said to be a top-down approach (from abstract legal concepts identified by experts, to their concretization in the legal documents under examination). The top-down approach is more complicated to accomplish, because it requires a domain expert. Furthermore, the level of abstraction required for top-down ontologies in legal domain may be challenging and time-consuming even for the best legal experts. On the other hand, the bottom-up approach is much easier to automatise, but it is prone to mistakes and redundancy, often producing worse results with respect to the other approach. This is why we propose to exploit the best of these two approaches by using a sort of ontological hinge that should be able to connect a bottom-up KG with top-down ODPs. In order to obtain this ontological hinge, we have to abstract new relations between the concepts of the ODPs and those of the extracted KG.

Question Answering
KG-based question answering consists in answering natural language questions about information contained in the KG. Let C be the set of concepts in a question Q. We perform KG question answering by: 1. Extracting: extract C from Q, using the same procedure adopted for extracting concepts during the KG extraction in section 3.1. 2. Matching: find the most syntactically similar KG concepts to C, and retrieve all their related template-triples including those of the sub-classes of C. 3. Selecting: among the natural language representations of both the retrieved triples and their respective subjects/objects 7 , select those snippets of text that are sufficiently likely to be an answer to Q. 4. Answering: return as set of answers the contexts (the source paragraphs) of the selected snippets of text (triples or simple concepts).
More in detail, the matching phase is performed by computing the similarity between the labels of every concept in the KG and every concept in C; we do it by using the algorithm described in [14]. Similarly to the matching phase, the selection phase is performed by means of a variation of [14], that combines Term Frequency-Inverse Document Frequency (TFIDF) with a version of the Universal Sentence Encoder (USE) for QA [16]. The main difference of the selection phase with the matching phase is that the similarity is computed between the questions and the contextualized triples/concepts in the KG. Every triple/concepts is represented in natural language (as in the matching phase) and its context is the snippet of text (the paragraph) from which the template has been originally extracted.

Related Work
In literature we found many works on Question Answering (QA), only few of them [18,17,3] were on Knowledge Graphs (KGs) and all of these were about RDF or similar technologies. As comparison to our work, we point to the many state-of-the-art deeplearning based QA algorithms implemented by Wolf et al. [15]. With these algorithms, using the whole Rome II Regulation EC 864/2007 as input context would require an impractical amount of time 8 for every posed question, in order to obtain very short (e.g. 2-3 words) answers which quality heavily depends on the selected linguistic model. The practical advantage of our approach over the others is that it is capable of selecting the most relevant text fragments in the context, limiting the search for an answer to very few paragraphs rather than the entire corpus. Furthermore the matching criterion [14] we adopted for answer selection combines both statistical and deep learning approaches trying to take the best from both, making the answering process a little bit more transparent.

Evaluation
We are interested in evaluating the usefulness of the resulting Knowledge Graph (KG), extracted from contract regulations, with respect to the legal user's needs. The goal in this specific domain is to extract knowledge according to specific situations and to detect the useful legal sources capable to help the expert to interpret them and to find a solution. A user can interact with the KG through the Question Answering (QA) tool, posing natural language questions and expecting useful answers from the system. Some frequently asked questions, in these cases, might be related to where a legal trial is celebrated (e.g., the pertinent jurisdiction and court), because there are many nuances and conflicting rules depending to the typology of actors, the country of residence (e.g., habitual residence), the country where the activity is performed (e.g.,country where the employee habitually carries out his work). For this evaluation we focus our attention on the jurisdiction and the judge a quo 9 . The adopted methodology comprises a team of legal experts selecting 8 relevant questions and evaluating the correctness of the answers provided by our algorithm. As shown in table , the resulting answers are in some cases imprecise, but overall the algorithm we described in this paper achieved an average top5recall 10 of 34.91%. Considering we have been using generic QA models (not fine-tuned on the specific domain) and a naive approach to paragraph matching (see Section 3.4; 8 Try it: https://huggingface.co/models?filter=question-answering 9 The judge a quo is the judge that is pertinent in the first grade of judgment and consequently the definition of this element is related with the jurisdiction applicable and the normative system valid 10 Let n be the number of strictly-correct produced answers, let |E| the number of expected answers for a question, then the top5-recall is   Table 1. Questions, expected answers and the top5 produced answers and their top5-recall -"B" stands for Brussels, "RI" for Rome I and "RII" for Rome II. "Rec." stands for Recital and "Art." for Article. The percentages shown are an estimate of the answer pertinence (the answer similarity defined in Section 3.4).

Question
Produced it could be improved by integrating information coming from an external reasoner), we believe the results are promising. As we can see the QA system is able to identify plausible answers for all the questions, even if they are clearly limited to the knowledge explicitly mentioned in the regulations. In many real-case scenarios we need to codify also implicit rules that are coming from the legal experts, in order to include also non-written relationships coming from the theory of law. In any case this approach compared with pure a full-text method is producing better results. Despite this, the results show that the QA algorithm is poor in reasoning (especially multi-hop reasoning), being trained to solve tasks related to commonsense, hence pointing to future developments.

Conclusions
This paper presents a hybrid and innovative approach to model legal knowledge extracted from heterogeneous legal sources, using ontology design patterns as skeleton for mapping the information deducted using OKE and linguistic NLP analysis. In a legal domain, with multiple conflicting norms and a large number of multiple definitions for the same concept, our proposed approach gives interesting results, providing a KG where the legal expert can easily retrieve the relevant information via critical queries. The KG provides a useful instrument for information navigation, that could be integrated in traditional information systems and legal databases. The confidence scores of the preliminary results are not optimal, but in the light of conflicting norms this approach could be an interesting outcome, in any case, because it integrates the legal interpretation methodology provided by the legal experts. We definitely need more testing with the help of the legal experts in order to tune the resulting pipeline defined in Section 3, but we believe that the proposed approach is correct especially in the domain where there is no unique accredited interpretation and the application of the norms depends too much on the hierarchy of sources. In the future we intend to integrate our approach in existing tools 11 for legal document analysis, as the first part of a sophisticated explanatory tool for making sense of complex legal documents, facilitating the process of representing legal knowledge in machine-compatible ways (e.g. ontologies, taxonomies, thesauri, etc..).