{'fraction_non_alphanumeric': 0.026357435197817188, 'fraction_numerical': 0.005675306957708049, 'mean_word_length': 5.090395480225989, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Corpus Query Lingua Franca (CQLF)

The present paper describes Corpus Query Lingua Franca (ISO CQLF), a specification designed at ISO Technical Committee 37 Subcommittee 4 “Language resource management” for the purpose of facilitating the comparison of properties of corpus query languages. We overview the motivation for this endeavour and present its aims and its general architecture. CQLF is intended as a multi-part specification; here, we concentrate on the basic metamodel that provides a frame that the other parts fit in.


Introduction
The present paper describes Corpus Query Lingua Franca (ISO CQLF), a specification designed at ISO Technical Committee 37 Subcommittee 4 "Language resource management" for the purpose of facilitating the comparison of properties of corpus query languages. We overview the motivation for this endeavour and present its aims and its general architecture. CQLF is intended as a multi-part specification; here, we concentrate on the basic metamodel that provides a frame that the other parts fit in.

Aims and Motivation
A lot of effort has been devoted to developing various corpus query languages (cQLs) in order to adequately satisfy information needs of corpus linguists and lexicographers. In practice, this large amount of different cQLs (see e.g. Clematide 2015) poses a serious challenge for researchers seeking to maximally exploit the advantages offered by contemporary language resources; this is true especially for those who do not have a background in computational linguistics and programming. Apart from learnability problems concerning some cQLs, the real problem is that a number of linguistic resources are accessible through drastically different corpus management systems which provide their own cQLs. Thus, mastering one or two cQLs is not enough to benefit from the entire palette of existing corpus linguistic software packages. Apart from that, as the respective fields develop, information needs grow and it may well happen that after having invested significant time and effort into learning and adopting a new cQL, a researcher will come to realise that the given cQL provides no way to further elaborate on the existing queries, and that another cQL has to be used.
There exists a need for interoperability across corpus query systems, realized among others by abstracting away from individual cQLs and seeing how far their queries can be compared. The issue of general interoperability of language resources and applications has been addressed at various levels of abstraction by such initiatives as e.g. CLARIN "Federated Content Search", ISOCat (now in redesign), or WSEP (Ide, 2013). With the purpose of enabling interoperability of corpus query tools (or at least gauging the extent to which such interoperability is possible), we have initiated the development of a standard for the presentation of features of corpus QLs, called Corpus Query Lingua Franca (CQLF), within ISO TC37 SC4 WG6. Currently at the Committee Draft stage, ISO CQLF (ISO/CD 24623-1) is the first standardization activity in the domain of linguistic QLs. 1 The goal of CQLF is to provide both a metamodel that would serve as a target space on which individual QLs can be located with respect to their properties, and a set of well-defined properties which could be applied as a measure of compatibility and interoperability between cQLs. CQLF is not expected to serve as an interlingua and provide the technical basis for query translation from one QL into another. Such initiatives raise a host of problems, ranging from low-level technical descriptions (e.g. the inability to preserve information when translating between regular expressions on the one hand and wildcards on the other) to issues of epistemology ("Does the result of the query reformulated in cQL2 address exactly the information need expressed in cQL1?"). The immediate goal of CQLF is therefore more modest: to serve as the target space within which QLs can be located with respect to their basic properties. It can thus also serve as a measure of compatibility and interoperability, but without the added claim to provide QL-to-QL mappings. A robust CQLF-based mapping system in action (with the epistemological burden appropriately controlled) would be ideal in the long run, but it makes a lot of sense to start smaller. CQLF is also expected to be useful to corpus linguists and Digital Humanists at large for the purpose of identifying the QL suitable for the corpus linguistic tasks that they are faced with.

Background
The background of CQLF has been framed by three sources that we describe in turn. A statement of existing demand has been provided by Martin Mueller's report for the Mellon Foundation "Towards a digital carrel: A report about corpus query tools" (2010) providing the general idea and the name of Corpus Query Lingua Franca. As Mueller noted in his summary, "it seems worth doing" to put "together an ISO proposal for a corpus query lingua franca that would allow different query engines to interoperate" and to make "recommendations for representing queries. If it leads to a standard, it will make development easier. If it does not lead to a standard it may at least help articulate the points where interoperability becomes difficult or breaks down" (Mueller, 2010). While the present proposal is crucially not a proposal for a single query language "to rule them all", we consider it an expression of what is feasible to achieve in this area, in terms of facilitating interoperability and highlighting the points where interoperability should not be expected. Subcommittee 4 of the ISO Technical Committee 37 (TC37 SC4), where CQLF was successfully accepted as a New Work Item and gradually developed into Committee Draft provides the institutional background for this initiative. A group of international experts have participated in the standardization process contributing to the creation of the present proposal. The practical background for experimentation on implementing CQLF has been supplied by the Leibniz-Association-funded project "Korpusanalyseplattform der nächsten Generation" at the Institute for the German Language in Mannheim (IDS Mannheim) where a new corpus analysis platform KorAP has been developed (cf. Bański et al., 2012;Bański et al., 2013, Diewald et al., 2016. At an early stage of the development of KorAP, a sizeable collection of use cases for querying corpora was compiled, and a number of cQLs were evaluated against it. That has provided a view on limitations and strengths of existing cQLs, and helped to identify the differences between them. The result of this work was one of the sources used to bootstrap the design of CQLF (see also Frick et al., 2012). Furthermore, part of the KorAP project focused on integrating CQLF concepts into a query management system, by providing a facility to translate the compatible corpus queries in various cQL into a common protocol, uniformly processed by the search backend. This in fact yielded the first reference implementation for CQLF (for more details see Bingel and Diewald, 2015).

Architecture of the Standard
CQLF is intended to eventually consist of three parts: • Part 1, "Metamodel", overviewed here, provides the specification of the abstract CQLF metamodel and circumscribes the outer limits of QL compatibility. • Part 2, "CQLF Ontology: Single-stream Architectures", describes the set of properties allowing to classify query languages within CQLF; this part has been implemented as an OWL ontology and is currently at the alpha stage. • Part 3 "CQLF Ontology: Multi-Stream Architectures" is intended to address the tasks of querying multiple primary data streams, for example in multi-modal corpora and parallel corpora.

Components of the CQLF Metamodel
CQLF is designed to be a modular construction with several components. Each component is characterised with respect to some aspect of the data models describing corpus objects that are the target or context of queries. A schematic view of the components of a CQLF metamodel is presented in Figure 1 below. The top-level components are referred to as CQLF Classes and correspond to the major division into data models built upon a single data stream vs. those which use more than one data stream (be it binary or text-based), in parallel. The present paper focuses on the Single-stream class (to be introduced below), which consists of three CQLF Levels that correspond to the major kinds of data organization in linguistic corpora.
The index variables of CQLF Level and Module are convenience details that allow for easier reference. The identification of levels and modules, while grounded in formal properties of data models and in functional characteristics of query languages, is also partially utilitarian, where it neglects some distinctions that could otherwise be recognised, in order to provide a simpler mechanism for determining conformance with the overall model and for stating the most important similarities and differences between cQLs. The relationships sketched in Figure 1 together with the basic divisions made on the basis of the existing cQL, yield a basic taxonomy of cQLs represented in the diagram in Figure 2 below.
Within the Single-stream class, each consecutive level introduces a more complex search dimension. The level system is based on the distinction between different major types of data organization and consequently different types of annotations (modelled by various parts of the ISO LAF family of standards), as well as queries that correspond to them. Below, we look at each level in turn. Level 1 (Linear) addresses, in any combination, plain-text search, search in segmented data (segmentation annotations) as well as search in simple annotations. What is relevant for this level is that annotations (if present) form a single layer of objects that exhaustively or partially describe the primary data stream. Tokenization is treated as a privileged level of segmental annotation (see (Odebrecht et al., submitted) for arguments for this point of view). Level 2 (Complex) consists of three modules. The Hierarchical Module concerns tree-based representations (primarily used for phrase-structure description). The Dependency Module focuses on the identification of relationships in which words function as nodes. The two modules can be composed into a mixed representation, with dependency relationships imposed on phrase structure. The third module of this level concerns simplified hierarchical relationships encoded as span containment. This can mimic phrase structure (with extra devices for resolving cases of mutual containment, from which the hierarchy may not be derived) and is often nonrecursive (as illustrated by e.g. CQP, cf. (Evert, 2010)). By definition, hierarchical and dependency annotations involve spans (anonymous or with simple annotations), either as terminal leaves in trees, or as objects over which relations are defined. Similarly, in order to create representations in which annotations of the same data stream are aligned (or contrasted), there must exist some kind of span (even if sentence-or paragraph-sized) that is a member of the relevant relation. Levels 2 (Complex) and 3 (Concurrent) therefore assume the existence of segmental annotations (and by a minimal extension simple annotations), and additionally allow for plain-text search. Level 3 (Concurrent) consists of two modules: Paradigmatic, responsible for prototypical cases where different annotation layers provide data packages competing for the same location, and Overlapping, which addresses concurrent annotations that are built upon spans which differ in their start and/or end offsets. Note that by assuming that concurrent annotations involve a single underlying data stream, the possibility of representing alignments by means of ordered sets of offset anchors (because each case of alignment is measured at a single offset in the single underlying data stream) is effectively eliminated, and reliance must therefore be placed on a subset of a simple span algebra, with values restricted to "partial/complete/no overlap". This is different from cases where multiple streams of primary data are aligned, and where ordered pairs or n-tuples of offsets, pointing into each of the parallel data streams, must carry important information. Such cases are not in the scope of the present specification and are the subject of CQLF Part 3, "Multi-Stream Architectures". The target data models are, minimally: The above requirements are stated as the baseline for assessing the position of the given cQL within the matrix proposed here. Since CQLF is meant to be maximally inclusive, the references to ISO-defined data models are provided as hints rather than preconditions. Note that in order to keep the number of levels small, some concessions have been made: plain-text search has been classified together with search within segments (whether token-level or larger) due to the fact that some matching mechanisms and syntactic operators are shared between the two. Similarly with search in simple annotations. In the same vein, hierarchical and dependency-based description again share many mechanisms (with mixed models well attested: for example, the Tiger XML (König et al., 2003) model is hierarchy-based, with "secondary edges" introducing dependencies between constituents; etc.). Containmentbased relationships are quasi-hierarchical (hierarchy can in most cases be derived from containment). Both Level 2 (Linear) and Level 3 (Concurrent) depend on simple annotations (alternatively, on segmental annotations), hence on Level 1. Level 3 may, but does not need to, build on Level 2. These interdependencies are illustrated in Figure 3.
2 Note that ISO LAF does not provide recommendations for putting together concurrent annotations other than by creating a super-graph over the graphs describing individual stand-off annotations.

Conformance
The aim of the conformance statements for the CQLF Metamodel is to classify the existing corpus QLs into major categories, corresponding to the CQLF components (classes, levels and modules). This classification is intended to be maximally inclusive and coarse-grained, and is a prelude to a more thorough feature-by-feature classification in the forthcoming parts of the standard. In order to claim conformance with the CQLF Metamodel, it is enough for a QL to qualify as conformant at any leaf node of CQLF component hierarchy illustrated in Figure 2, as long as the following conditions are also obeyed: • In order to be able to claim conformance at Level 1 (Linear) of the CQLF Metamodel, a corpus QL must provide support for plain-text search and/or search in segmentation annotations and/or search in simple (non-hierarchical and non-dependency) annotations. • In order to be able to claim conformance at Level 2 (Complex) of the CQLF Metamodel, a corpus QL must provide support for querying hierarchical annotations and/or dependency annotations and/or annotations (segmentation or simple) related by a containment relationship, and it must be conformant at Level 1. • In order to be able to claim conformance at Level 3 of the CQLF Metamodel, a corpus QL must provide support for querying the relationships/alignment between simple annotations (minimally) or between a mixture of simple annotations, hierarchical annotations, and dependency annotations, and it must be conformant at Level 1.
It is essential to draw a distinction between CQLF conformance and CQLF-Metamodel conformance; the former must contain a list of references to the (sets of) features defined in CQLF Part 2: CQLF Ontology: "Single-stream architectures". This reference will be in the form of a list of Data Categories (or Data Category Selections) rooted in an instantiation of ISO DCR. Below, we list several self-documenting examples of conformance statements, extracted from the CQLF ontology.
(1) Single-stream → 1. Linear → plain-text Note that the present specification does not specify conformance conditions for CQLF Class "Multi-stream". These will be the subject of Part 3, "CQLF Ontology: Multi-stream architectures".

Summary and Outlook
The present paper presented an overview of the architecture of CQLF and sketched its further projected development. CQLF is designed to provide a useful basis for establishing the potential extent and the limits of interoperability between different corpus query systems, and to help reduce the gap between end users with a linguistic or literary background and powerful search environments using modern technology. 3 The aim is to provide a metastandard that will form the basis for locating any individual corpus QL within a single matrix of a few well-defined properties, and then, gradually, to elaborate on, and extend, the inventory of these properties, as well as to define the relationships between them. An initial set of properties has been designed according to both formal and functional criteria, and may be used as guidelines in the development of a new QL or a new functionality of an existing QL. These properties are part of the 2nd part of CQLF, "CQLF Feature Ontology" (implemented in OWL and currently in the alpha stage), which includes a relatively small ontology to be embedded in an infrastructure that will allow for extensions by individual cQL developers, and for look-ups by end users. Figure 4 illustrates the CQLF infrastructure that we envision, with two major entry points: one for developers entering information on their cQLs in order, among others, to claim conformance with CQLF, and the other for end users seeking to identify the range of cQLs that satisfy their requirements.

Acknowledgements
We would like to acknowledge the helpful remarks that we have received from the DIN mirror of ISO TC37 SC4, and in particular Stefan Evert and Kerstin Eckart. We are also grateful to Michael Cunningham for his patient assistance with making the specification more accessible, and to Guadalupe Aguado de Cea for inspecting the set of proposed definitions. We wish to stress that all possible errors introduced at later stages of development are our sole responsibility. Finally, we wish to heartily thank three anonymous LREC reviewers for encouragement and useful remarks.