{'fraction_non_alphanumeric': 0.025855048859934854, 'fraction_numerical': 0.00760043431053203, 'mean_word_length': 5.22581326573722, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Provenance for Linguistic Corpora Through Nanopublications

Research in Computational Linguistics is dependent on text corpora for training and testing new tools and methodologies. While there exists a plethora of annotated linguistic information, these corpora are often not interoperable without significant manual work. Moreover, these annotations might have adapted and might have evolved into different versions, making it challenging for researchers to know the data's provenance and merge it with other annotated corpora. In other words, these variations affect the interoperability between existing corpora. This paper addresses this issue with a case study on event annotated corpora and by creating a new, more interoperable representation of this data in the form of nanopublications. We demonstrate how linguistic annotations from separate corpora can be merged through a similar format to thereby make annotation content simultaneously accessible. The process for developing the nanopublications is described, and SPARQL queries are performed to extract interesting content from the new representations. The queries show that information of multiple corpora can now be retrieved more easily and effectively with the automated interoperability of the information of different corpora in a uniform data format.


Introduction
The availability of annotated linguistic corpora is crucial to develop Natural Language Processing (NLP) systems [5,25]. In the past couple of decades, many linguistic resources have been released, which contain different types of annotations, from morphosyntactic to semantic annotations [5]. Often, such annotation projects run on (subsets of) existing text corpora, which are thereby annotated with multiple annotation layers. Unfortunately, the format for linking annotations to the texts is mostly not standardized and these links therefore are often not fully precise or directly interoperable. This makes it challenging for a researcher to figure out the data's exact provenance and to integrate it with annotations from other corpora. Moreover, often the annotations follow idiosyncratic guidelines that pursue a specific annotation goal. Integrating different annotations is valuable because it can help scholars study and understand the different interdependencies between annotation layers (e.g. semantic parsers usually take syntactic structure in consideration) and get a broader understanding by combining different annotation types (e.g. how events from event annotations are linked to persons from named entity annotations).
The need to organize the various existing annotations has been recognized by the corpus linguistic community [5,10]. While a number of solutions to improve linguistic Linked Data have been presented in the past, we propose here to use a novel fine-grained and flexible Linked Data format based on nanopublications to represent annotated corpora with the aim of boosting the interoperability of linguistic corpora and facilitating the automatic integration of annotations in a reliable and transparent manner. As a case study, we have chosen to focus on integrating annotations from event corpora, because the corpora are widely used in the NLP community and there are many different types of event annotations that continue to be made on the same corpora. The corpora selected for this project are FactBank [24] and PARC 3.0 [19]. We describe below how we converted these corpora into nanopublications and then report on the interoperability challenges faced and solved with our approach. Finally, we discuss based on our results the extent to which nanopublications can be seen as a viable alternative for improved linguistic corpora interoperability.
Our main contributions thereby are (1) providing a fine-grained and interoperable Linked Data representation of annotated linguistic corpora in the form of nanopublications, (2) analysing the main interoperability problems that we encountered and solved with our nanopublication-based approach, and (3) demonstrating how the concrete annotations from our case study of two corpora can be integrated with nanopublications and queried with SPARQL.

Background
Below, we introduce the relevant background on the interoperability of linguistic corpora with a special focus on event annotations, Linked Data and data provenance, and the specific concept and technique of nanopublications.

Linguistic Corpora Interoperability and Event Annotations
Many types of annotated linguistic corpora exist today and are available for scientists to use, but they often lack structural and conceptual interoperability [5]. The former is related to the data format of annotations, such as XML-standoff or RDF representations, while the latter refers to the possibility of exchanging information in a consistent and mappable way. Diverging annotation guidelines, the structure of annotated output, and the different amounts of annotations created per corpus pose a challenge in organizing the provenance of corpora.
As mentioned, this project focuses on event annotations. In corpus linguistics, events are generally considered as situations that can happen or occur [20]. However, existing event annotation standards range from annotations about the participants, timing/temporal order, factuality, and entity coreference relation [25], making a standardized description of an event difficult. For example, Figure  Fig 1 presents the same sentence annotated in PARC and FactBank. In PARC, attribution relations and their components (Cue, Content, Source) are annotated, while in FactBank events and their factuality values are annotated (hurt, disposal, restructuring, said ). It is interesting to combine the two annotation layers in order to extract, for example, which events occur in the content of attributions relations by the same source and what are their factuality values.
Our research is motivated by the empirical analysis on 20 event annotated corpora presented in [25], who highlighted the challenges, as well as the opportunity, of event corpora interoperability. This is where nanopublications as a fine-grained and provenance-aware data format could provide transparency and interoperability for computational linguistic research.

Linked Data and Data Provenance
Linked Data is about connecting and publishing data in a structured way on the web, often in the form of an ontology using the Web Ontology Language (OWL) [17]. The data is often structured in triples using the Resource Description Framework (RDF). By using unique URIs for each entity of information, these entities can be interlinked in order to create an entire web of data [1].
Several projects were already performed to create linked linguistic data. Since 2012, six series of Linked Data in Linguistics (LDL) workshops have been hosted to gather and discuss contributions promoting linking linguistic data. After the second workshop, one of the presented efforts was the Linked Linguistic Open Data (LLOD) cloud 3 . While creating the LLOD cloud, a lot of linguistic data was already available, which were mostly represented using RDF. One example is WordNet, which is also part of the Semantic Web [8,6]. Furthermore, Chiarcos proposed to represent linguistic corpora into OWL and RDF to interlink all of the different resources of corpora. In this way, the annotations of the corpora could be linked using terminological resources like standardized annotation formats [5].
Existing approaches that aim to represent the structure and content of natural language and it annotations include the Ontology Lexicalisation and the Ontologies of Linguistic Annotation (OLiA) [26,4,2]. OLiA focuses on annotations of linguistic corpora and can be seen as a source that references to the 'annotation terminology' that can be used in combination with the LLOD to capture all of the information of the annotations in the LLOD in a similar format [7].
Since all different texts and annotations in a corpus may have a different origin, i.e. provenance, provenance ontology PROV 4 can be used for this. This ontology contains different classes and properties that represent the provenance information in different contexts, thereby usable for keeping track of the provenance information of linguistic corpora. The PROV ontology has been shown to be applicable to a diversity of fields to track provenance information in a general manner [16].
Even though the RDF language is not optimized for this, the NLP Interchange Format (NIF) 5 makes it possible to represent texts and their elements in RDF, primarily by using strings. With the NIF ontology, a context, sentence, phrase, or word can be represented into RDF triples. It is also possible to have references between different levels, so a word can reference to another part of the RDF scheme that is the sentence from where it was derived.
Moreover, many annotation guidelines exist and each is usually represented in a different structure, increasing the difficulty in creating a standard model to transform annotations into RDF. The Web Annotation Vocabulary [21], for example, provides a possible solution with relations such as 'motivatedBy' and 'hasTarget', making it possible to structure annotations in an interoperable way.

Nanopublications for Linking Linguistic Corpora
Nanopublications are a format for small data publications based on RDF triples, consisting of three parts: the assertion, provenance, and publication information [12]. The assertion contains the main content of a nanopublication, for example stating in a formal way the link between a gene and a disease. The provenance part states how this assertion came to be, by linking to the study that was performed or the paper from which the assertion was extracted. The publication information part, finally, records information about the nanopublication itself, such as by who and when it was created [9].
In a previous work, a Java library for nanopublications was created, which can also be used as a command line tool [11]. Nanopublications can be given trusty URIs as identifiers, which include a hash value of the complete content, and thereby make nanopublications immutable and verifiable [14]. Such nanopublications can then be reliably and redundantly published to the existing distributed server network [13]. Each of these servers contains all nanopublications, making it possible to retrieve the nanopublication from another server when one server is down.
Nanopublications also allow for precise and reliable versioning, as well as the definition of incremental datasets by defining nanopublication indexes [15]. Such indexes are represented as nanopublications themselves that contain links to other nanopublications and thereby defining sets of nanopublications.

Methodology
Here we introduce our methodology to address the above-mentioned problems of interoperability and provenance of text corpora by applying the concept and techniques of nanopublications. To demonstrate and evaluate this methodology, we then present a case study on the two overlapping corpora mentioned above. We introduce a number of questions on the combined annotations that can be answered in an integrated and automatic manner through SPARQL queries.

Nanopublication Model for Text Corpora
In Figure 2, we present a general scheme for representing text corpora and their annotations as a network of interconnected nanopublications. This model ensures that new information from other corpora could easily be linked to existing nanopublications, through the addition of more nodes to the network. The nanopublications that form this general structure can be divided into four groups depending on what they represent: text, annotation, corpus, and index nanopublications.
First, at the top of the figure the text-corpus nanopublication can be found. This nanopublication contains the information regarding the text corpus, which is the set of original texts. It is linked to an index nanopublication which points to all document, text, and word nanopublications which are part of the corpus. The document and text nanopublications can be part of multiple corpora and thus multiple index nanopublications. These document nanopublications contain all of the metadata on the actual texts, namely the provenance information, title, and identifier. Furthermore, the document nanopublications have a link to the nanopublication containing the actual text corresponding to the metadata. Both the document and the text nanopublication also contain a back-reference to their corpus nanopublication.
Additionally, word nanopublications are created, as words are the typical targets of annotations. They contain the word string, sentence number, and the offset values with respect to the text, and possibly other word-level information (e.g. their part-of-speech value). Thereby, each word gets its own URI as identifier, which is constructed from the text URI and the word's offset values. From the bottom, annotation corpora are pointing to these words. Similar to the text corpora, a corpus nanopublication of an annotation corpus contains the general corpus-level information, most importantly a link to an index nanopublication pointing to all annotation nanopublications. They contain the actual annotations and their concrete content depends on the corpus, but they all point to the words they are annotating via the URIs as described above. For convenience, they are also pointing back to the overall annotation corpus nanopublication. Modeling these elements as small self-contained entities has also the advantage that differences in licenses can be defined and handled on a fine-grained level. Often, annotations have a different, sometimes more permissive, license than the texts they annotate. For this, we introduce the nanopublication type ProtectedNanopub that we use to mark nanopublications that cannot be openly published due to license reasons. We updated the nanopublication client and server code such that an error is raised if a user accidentially tries to publish such a protected nanopublication to the server network.
Lastly and importantly, our approach is open for anybody to publish further annotations or texts as nanopublications. These wouldn't be directly included in the datasets, as the index nanopublications don't point to them, but they can be found by querying the nanopublication network, and included if desired in a given situation. Moreover, a new version of a corpus can include such "third-party" contributions, thereby making them a proper part but also clearly attributing the third-party contributor, which is defined in the provenance and publication info parts. This also allows for corrections to be published by anybody who finds something that is wrong.

Case Study Design
The case study workflow has 3 main components: (1) obtain and analyze original and annotated corpora; (2) convert these into nanopublications; and (3) assess  [23], while it defines event factuality with a value that represents how certain or true an event is according to a source present in the text. FactBank has six primary factuality values present in its annotations. These annotations are contained in a set of 20 tables, each written in a separate text file. We refer the reader to [22] and [18] for detailed explanations of each annotation scheme, and we will elaborate below on their interoperability challenges.
For this case study, we came up with six general questions that somebody who would like to use the above annotations might want to ask. These questions are shown in Table 1. One of our goals is answering questions that show the merging capabilities and efficiency of nanopublications. For example, the answer to Q3, depending on the factuality values it outputs, might suggest how certain the content of the text is. An answer to Q6 might help an annotator clarify doubts during the annotation process, and Q5 demonstrates how FactBank events can be related to PARC attribution relations. Some questions ask only for information from one corpus, while others are more complex by requiring information from both corpora. We will use these general questions to make them automatically executable with our integrated nanopublication model and the SPARQL query language.

Case Study
We applied the nanopublication-based approach to the case study in order to address the problem of interoperability and provenance. We discuss here the different types of nanopublications created to represent the chosen corpora.
As introduced above, nanopublications consists of an assertion, provenance, and publication info part. The publication info part is similar for all our nanopublications, containing the date of creation, license, ORCID identifiers of the creators, and sometimes further comments and website links. All the text and word nanopublications, as well as the annotations of FactBank, are also marked as protected, due to their private licences. The remaining nanopublications are public and can therefore be published to the server network for public accessibility. The dataset of all public nanopublications can be found online. 6 We defined four corpus nanopublications for our case study: one for the text contained in the corpus and one for the annotations contained in the corpus, for each of the two existing corpora (PARC and FactBank). They contain basic information about the corpus and link to their content via an index nanopublication, as can be seen in this example (abbreviated here and below for readability): The word nanopublications point back to the text nanopublication and provide word-level information including the word string, the offset of this string with respect to the text, and the sentence number. They can also contain lemma and part-of-speech information. As this word level depends on tokenization, there can be differences in how different annotation corpora define word boundaries.
For this reason, our approach allows for the generation of new word nanopublications on demand. This creates an identifier for the word based on the URI of the text nanopublication and the offset of the word, thereby assuring that words introduced several times by different annotation corpora get the same identifier and are therefore immediately interoperable. In our case, we have 6,362 word nanopublications defined from PARC and 7,784 from FactBank. This is an example of the former: sub:assertion { textoffset:3-9 a nif:OffsetBasedString, nif:Word; nif:beginIndex "3"^^xsd:int; nif:endIndex "9"^^xsd:int; nif:anchorOf "ROGERS"; nif:lemma "rogers"; olia:POS "NNP"; pvcp:hasSentenceNumber "0"^^xsd:int; pvcp:isPartOfText textnp:text . } The annotations can now link to these words. We have 136 annotation nanopublications for PARC, that associate words with content, cue and source annotations, as can be seen in the example below: For FactBank, we have annotation nanopublications that declare events (7,784) and separate ones annotating their factuality values (10,948). Below is an example of an event declaration, and another example of a factuality value annotation (the factuality value is here "CT-") linking to such an event declaration: We have focused above only on the assertion part, but all these nanopublications also come with specific provenance and publication information, for example describing that the PARC annotations were created at a different time by a different person than the nanopublications themselves: sub:provenance { sub:assertion dct:created "2016-05-01T00:00:00"^^xsd:dateTime; dct:creator orcid:0000-0003-0955-6104 . } sub:pubinfo { this: dct:created "2020-05-29T11:53:00.700+02:00"^^xsd:dateTime; dct:creator orcid:0000-0002-3429-2879, orcid:0000-0002-5347-5750, ... . }

Results
We can now analyze the interoperability challenges we faced and addressed with our approach and assess the extent to which we can now automatically answer the questions we introduced above.

Analysis of Addressed Challenges
With our case study as described above, we encountered a number of interoperability challenges. We managed to resolve almost all of them in our manual conversion work as described above. However, we had to exclude ten documents of FactBank due to the fact that sentence numbers were skipped when a sentence ended with a double quote ("), and we did not have enough information to correct this mistake. In Table 2, we show an overview of the challenges we successfully addressed on the remaining 194 documents. This highlights the nature and extent of interoperability problems that come with the current way how such corpora are published but would be resolved if our approach was followed, i.e. if they would be published as interoperable nanopublications from the start.
The first issue on the list are text offsets. PARC and FactBank locate words in a text differently: PARC uses byte counts, whereas FactBank makes use of sentence and token numbers. Token numbers are tokenizer-specific, so FactBank's method is quite sensitive and difficult to replicate. FactBank's sentence numbers are moreover confusing as they also count the metadata fields, such as text headings, and not just the text content (which is challenge number 2). PARC, on the other hand, also has its quirks, such as including the six characters of the initial <TEXT> tag in the offset count. In our nanopublications, we use text offsets that unambiguously link to the text string of the content, and thereby ensure precise and interoperable text references. For FactBank's sentence and token number approach to refer to words in the text, it is crucial to know where a sentence ends and a new one starts. This sentence splitting, similar to tokenization mentioned above, is sensitive to the used algorithm. This is why text corpora like WSJ provide us with this sentence splitting information by saving each sentence on a separate line in the provided file formats. However, the documents from NYT and APW didn't have this information and so these sentences had to be checked manually for the correct splitting (challenge 3). Moreover, sentences containing a semicolon in PARC are annotated as one sentence while they are considered two sentences in FactBank (challenge 9).
Challenges 5 and 6 point to the fact that text tags were used inconsistently. The main text is included in <TEXT> tags, but these were missing for 17 documents. Moreover, WSJ documents use <HL> and <DATELINE> tags for the headline and date, respectively, while NYT and APW documents use <HEADLINE> and <DATE _ TIME>. Some documents moreover, did not contain tags at all, and we had to add them manually.
Normally, headlines are not annotated, but there is an exception in document APW19980213.1380, which includes an annotation of a part of the headline (challenge 10). Upon encountering this situation, we extended our model to also allow for headline annotations. Lastly, in some cases metadata was missing. Some documents did not have a headline (challenge 4) or did not include a journal or source (challenge 7). In PARC, there were also four documents that did not contain any attribution relations (challenge 8).
In summary, all documents were affected by at least two challenges, and there seems to be a long tail of infrequent exceptions or small inconsistencies that one has to take into account in order to process such corpora correctly.

Query Results
In order to assess the practical benefits of our approach we implemented the general questions presented in Table 1 as queries in the SPARQL language so we could use a triple story (Virtuoso in our case) to automatically answer them. The resulting SPARQL queries can be found online 7 . They demonstrate that our nanopublication approach effectively merges the two corpora and allows for information to be extracted together in a practical and efficient way. Table 3 shows the average amount of time it took to run a query, as well as the number of rows it outputs. We managed to represent all these questions in SPARQL, and thereby to make them automatically executable. The listing below shows an example of a SPARQL query to answer one possibility for question Q4 ("How often are certain annotation values used?") retrieving the count of words annotated as events from FactBank per PARC annotation attribution type: This shows again how the annotations from both corpora can be queried simultaneously. The shown portion of the result is very informative for the file wsj _ 0026. The content of the columns, in order of appearance from left to right, is: the text identifier in the form of its nanopublication URI with the suffix #text, the event identifier of the event annotation, the word that is annotated as the event, the factuality value of the event, the source of the event (according to FactBank annotations), and the string in the document's text that represents the source (according to the source annotated words in PARC). As such, the query's result contains information from PARC (the source) as well as information from Fact-Bank (the factuality value and event). Moreover, the output of the query allows us to see that an event can have two different FactBank sources who each give a different factuality value to the annotated event. This is seen in the last two rows where the event 'beneficiaries' has two different factuality values, one which was given by the AUTHOR (the writer of the text document) and another given by the 'officials' (a source talked about in the text document). The different conception of 'source' in FactBank and PARC can also be noted here, which shows that although annotations can be merged, the annotation schema of each corpus should always be considered when analyzing the results. This is also useful to reveal the representation of a corpus's data more easily. For example, we can see that an event from FactBank always has an AUTHOR source.
This demonstrates how we can use the power of query languages like SPARQL to access corpora in an integrated and fully automated way. The results can be exported in a variety of formats, including CSV tables and JSON files, and be loaded and processed in other tools. It is then straightforward to write conversion tools to other formats, such as the CoNLL format [3], a common practice data format used by computational linguists. With our nanopubliation approach, we ensure that the data is interoperable from the start, and then format conversions are relatively easy as the range of nasty interoperability challenges is already taken care of.

Discussion and Conclusion
Nanopublications have been mostly used with scientific data, such data from the biomedical domain [12]. In this work, we show that linguistic corpora can also benefit from the fine-grained and provenance-aware structure of nanopublications. The collection of nanopublications presented here contains information from event annotated corpora. It is demonstrated that when this data is modelled in a homologous way, each unique corpus can be automatically merged in order to produce valuable linguistic data combinations or informative corpus content that was not easily attainable prior to its transformation. Specifically, our dataset can provide attribution annotation insight about events from Fact-Bank.
Moreover, our nanopublications suggest the prospect of linguistic data fitting into the Linked Data ecosystem. During our dataset creation, we were able to reuse multiple existing Linked Data vocabularies. Linguistic data becoming more linked would also allow it to be connected more easily with already existing linguistic Linked Data sources from LLOD. Nevertheless, the data format of the original annotated corpora is extremely variable, which shows in the multiple textual challenges faced during production. While this project only focused on two corpora, there was inconsistency in annotation format not only between the corpora, but also within a corpus itself. Thus, the conversion of existing linguistic corpora into a Linked Data scheme can expect tedious pre-processing efforts. Even in our dataset, several documents needed to be excluded for the final version. Another aspect that makes it challenging to fit linguistic corpora in the Linked Data ecosystem relates to copyright availability. For example, this caused us to only be able to represent part of the PARC corpus.
In this paper, interoperability issues in linguistic data, specifically event annotated corpora, are discussed and nanopublications are proposed as a solution to resolve them. When one set of annotation guidelines is used to produce annotations on one corpus, that same corpus might already have other annotations that could be useful for the researcher. However, currently it is very difficult for a researcher to know or retrieve the version or provenance of these existing annotations due to poor documentation and variable annotation formats. Aspects of nanopublications that make them useful for annotated corpora are that they are able to be written in different formats, they can represent versioning, and they can track provenance. Also, they follow LLOD principles of using URIs for identification and RDF representation as output. Above all, they seem useful for merging linguistic annotation data. A successful model for translating existing event corpora into nanopublications is significant not only for computational linguistics, but also for the field of Linked Data.