{'fraction_non_alphanumeric': 0.03570833564724829, 'fraction_numerical': 0.017604265008052424, 'mean_word_length': 4.958967571144937, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Extended DIRNDL Corpus as a Resource for Coreference and Bridging Resolution

DIRNDL is a spoken and written corpus based on German radio news, which features coreference and information-status annotation (including bridging anaphora and their antecedents), as well as prosodic information. We have recently extended DIRNDL with a fine-grained two-dimensional information status labeling scheme. We have also applied a state-of-the-art part-of-speech and morphology tagger to the corpus, as well as highly accurate constituency and dependency parsers. In the light of this development we believe that DIRNDL is an interesting resource for NLP researchers working on automatic coreference and bridging resolution. In order to enable and promote usage of the data, we make it available for download in an accessible tabular format, compatible with the formats used in the CoNLL and SemEval shared tasks on automatic coreference resolution.


Introduction
The Discourse Information Radio News Database for Linguistic analysis (DIRNDL) is a spoken corpus resource of German radio news (ca. 50.000 tokens, 3221 sentences). In its first release (Eckart et al., 2012), it was manually annotated for referential information status, i.e. the givennew classification of referring expressions (Riester et al. 2010) as well as prosodic GToBI(S) labels (Mayer, 1995). Constituent-structure annotations originated with the XLE parser (Crouch et al., 2011) and the LFG grammar by Rohrer and Forst (2006). Aligning spoken language with its written transcript (or text with one of its read realizations) in a single resource is challenging for several reasons. Obviously, speech has a temporal determination which written language lacks. Punctuation marks (e.g. decimal points/commas) and compound words may receive different tokenisations in the different processing pipelines for written and spoken language, respectively. Moreover, speech, even the one by trained newsreaders, is seldom flawless and contains disfluencies and slips of the tongue, which are not contained in the written transcripts. (If they were, this would cause trouble to the parser.) In DIRNDL, these problems are tackled by integrating both sets of data with their different tokenisations in a PostgreSQL database and providing an alignment that uses multiple links (e.g. for accidental repetitions). The database has proven to be a valuable resource for testing linguistic hypotheses at the interface between discourse, information structure, morpho-syntax and prosody. For instance, Riester and Piontek (submitted) extract all adjective-noun sequences from the corpus, together with their prosodic realization, in order to test whether NPs with accented adjectives necessitate the existence of contrastive alternatives. Augurzky et al. (submitted) investigate the influence of segmental clashes on the frequency of prosodic phrase breaks at the transition between two referring expressions, and between nominal heads and their embedded arguments. Rosenberg et al. (2012) and Soto et al. (2013) use DIRNDL for training an automatic prosodic labeler. We have recently improved DIRNDL by revising existing annotations, and by adding new annotation layers, e.g. constituent trees and dependency trees from Björkelund et al. (2013), and named entities using Finkel et al. (2005) and Faruqui and Padó (2010). Based on this extension, we extracted DIRNDL anaphora as a resource for evaluation of automatic coreference and bridging resolvers. We exported the corpus in three tabular formats, two from recent shared tasks on automatic coreference resolution, i.e. the SemEval 2010 (Recasens et al., 2010) and CoNLL 2012 (Pradhan et al., 2012) shared tasks, and a third tabular format containing additional annotation layers which are not represented in the CoNLL or SemEval format, but might be useful for the resolution task, e.g. information status labels and pitch accents. In this paper we describe the exported annotation layers as well as the formats used. The export is freely available for download. 1

Annotation layers
In this section we review the various layers of annotation in the DIRNDL corpus and the new export. Table 1 gives an overview of the annotation layers in the DIRNDL corpus as described by Eckart et al. (2012) and in the new DIRNDL anaphora release.

Pragmatic annotations
The corpus was previously annotated for referential information status following Riester et al. (2010). These annotations were replaced by two-dimensional informationstatus annotations following the RefLex scheme (Baumann and Riester, 2012), which distinguishes between referential and lexical information status. For example, (1) contains a referentially given (coreferential) phrase which comes with new lexical material (a so-called epithet). By contrast, in (2) there is a referentially new phrase which features lexically given material. For the referential as well as for the lexical level, the corpus contains links between the anaphor and its antecedent. DIRNDL DIRNDLanaphora (Eckart et al., 2012) Pragmatic annotations information status information status according to according to RefLex scheme Riester et al. (2010) (Baumann and Riester, 2012) Prosodic annotations GToBI(S) labels for pitch GToBI(S) labels for pitch accents and accents and boundary tones boundary tones (revised manual annotations) Morpho-syntactic annotations constituent trees by i) lemmas predicted XLE parser with by the Mate lemmatizer LFG grammar of (Bohnet, 2010) Rohrer andForst (2006) ii) part-of-speech tags and morphological tags by MarMoT  iii) constituent trees and dependency trees from Björkelund et al. (2013) Semantic annotations named entities predicted by the Stanford named entity recognizer (Finkel et al., 2005)  By referential information status we refer to the classical notion of information status discussed in the literature, e.g. Prince (1981), Nissim et al. (2004), Riester et al. (2010). A referring expression is R-GIVEN if it is a coreference anaphor. The label R-BRIDGING indicates that we are dealing with a bridging anaphor (Asher and Lascarides, 1998;Poesio and Vieira, 1998), i.e. a non-coreferring but nevertheless context-dependent expression, e.g. the European Union . . . [the member states] R-BRIDGING . Lexical information status captures semantic relations (e.g. a noun, verb or adjective is L-GIVEN if it is identical, a synonym or a hypernym of word contained in the context). An overview of the basic labels is shown in Table 2. These labels also have subcategories, and we refer the reader to Baumann and Riester (2012) for further details. Interannotator agreement on radio news data was determined in Riester and Baumann (2013) at κ = .75 for the referential level, and κ = .64 for the lexical level. For both levels of information status, Baumann and Riester (2013) show that increased givenness on both levels leads to a lower accent rate and/or the use of perceptually less prominent (e.g. L*) accents. In particular, it is well-known that (noncontrastive) coreferential anaphors in English and German are often deaccented, a fact which is well-described in the literature (see e.g. Halliday (1967), Schwarzschild (1999), Umbach (2002), Büring (2007) and many others). It is therefore likely that information about pitch accents will be a useful feature in coreference resolution.

Prosodic annotations: GToBI(S) labels
DIRNDL comprises information about intonation, i.e. the way an utterance is organized tonally. A group of intonation models -autosegmental-metrical models (essentially all based on Pierrehumbert, 1980) -is well accepted and widely used when describing prosody. For a subset of DIRNDL (approximately 5hrs of speech), tonal events were annotated manually according to an autosegmental intonation model for German (GToBI(S), cf. Mayer, 1995). Tonal events are pitch accents, which mark some of the words in a phrase as being more prominent than others, and boundary tones, which mark the tonal phrasing of the utterance. Essentially, a tonal event can be described as a local maximum or minimum in the intonation contour. Therefore, GToBI(S) labels describe the pitch contour by means of two levels, low (L) and high (H) representing regions in the speaker's register. That is, H describes a high local target (a peak) and L indicates a low local target in the contour. For example, a rising accent is composed of a low target on the accented syllable followed by a rise of the contour on the post-accented syllable, and is therefore labelled as L*H. Analogously, H*L marks a falling accent.
The GToBI(S) inventory also includes labels for the boundaries of tonal phrase: intermediate phrases, which are minor tonal phrases, are marked with the label "-", intonation phrases, which correspond to major tonal phrases, are marked with the label "%". The latter can also be marked with a tone if the contour rises, or falls, respectively, at the end of the phrase (H% or L%). Table 3 gives an overview of the complete label set.
Pitch accents are annotated on the syllable level. To make the annotations available on the word level, in DIRNDL each accent was enclosed in two "|" symbols and if several accents occurred on one word token, they were concatenated in the order of appearance. For example if a token on the word level was accented with a rising accent followed by a falling one, it is represented as |L*H||H*L| For DIRNDL anaphora , the GToBI(S) annotations were checked for plausibility using pitch accent shape information as retrieved by a parametric intonation model (Möhler, 2001), and corrected if necessary.

Automatic morpho-syntactic annotations
The DIRNDL corpus was originally parsed with the XLE parser (Crouch et al., 2011) and the LFG grammar by Rohrer and Forst (2006). XLE provides deep LFG constituent structure analyses 2 but unfortunately yields fragmented parses in a substantial number of cases. In order to provide additional, more robust syntactic information, as well as more fine-grained morphological annotations, we apply several other automatic tools to the corpus. Specifically, we added the following annotations: (i) automatically predicted part-of-speech tags and morphological tags, predicted with MarMoT , which has been shown to outperform other available partof-speech and morphology taggers; (ii) predicted lemmas, using the lemmatizer of the Mate tools toolkit (Bohnet, 2010), a state-of-the-art statistical lemmatizer; (iii) automatically predicted constituent trees with the constituent parser from Björkelund et al. (2013); (iv) automatically predicted dependency trees with the dependency parser from Björkelund et al. (2013). The constituent and dependency parsers by Björkelund et al. (2013) have shown state-of-the-art performance and recently obtained the best 2 LFG F-structures are currently not integrated in the database.
results for German in the recent SPMRL 2013 Shared Task on parsing of morphologically rich languages (Seddah et al., 2013). In contrast to the LFG parser, which is rulebased and driven by a grammar, all these tools are datadriven. They were all trained on the TiGer treebank (Brants et al., 2002;Seeker and Kuhn, 2012) and therefore provide annotations adhering to the TiGer annotation scheme. Example constituent and dependency analyses of a fragment of (1) is shown in Figures 1 and 2, respectively. Since we have no gold standard annotations for these layers we are unable to evaluate the accuracy of these tools on the DIRNDL data set, however we refer the reader to the papers of the respective tools for evaluations on other data sets.

Automatic named entity annotations
Named entities are closely related to coreferentiality. In the RefLex scheme, named entities typically receive an R-UNUSED label 3 on their first occurrence, and an R-GIVEN label on subsequent occurrences. We added named entities using the Stanford named entity recognizer (Finkel et al., 2005). Specifically, we used the German model created by Faruqui and Padó (2010), which, in addition to standard

DIRNDL export
This section describes the basic constitution of DIRNDL anaphora and gives an example of the tabular export format used in the release.

Constitution
The DIRNDL corpus consists of hourly radio news broadcasts from three days during 2007. The respective text transcripts were retrieved from the website of the corresponding radio station. The export of DIRNDL we describe in this paper does not contain the audio files of the spoken news, but is restricted to the transcripts. It is important to note that, since the news broadcasts were consecutive, several items are repeated across broadcasts, sometimes with minor changes in between. 4 When using this resource either for training or testing automatic systems, we advise users to pay attention to these repetitions while conducting their experiments.

Tabular format
The original representation of the DIRNDL corpus is a relational database. While a relational database enables elaborate SQL queries, interfacing with a relational database is not the most convenient approach for NLP developers that are working on training and testing automatic systems. We therefore provide the new DIRNDL export in a tabular format, similar to the one used in the CoNLL 2011 and 2012 shared tasks (Pradhan et al., 2011;Pradhan et al., 2012). This also means that existing evaluation tools for automatic coreference can be used off the shelf against DIRNDL anaphora . An example of the tabular format is given in Figure 3, representing the two sentences from (1). The format represents each token on a single line and sentences separated by blank lines. Document boundaries are represented by the lines #begin document and #end document, where the former also contains a document identifier. In addition to the surface forms of each token, annotations are provided as additional columns in each line. A summary of the contents of the columns is displayed in Table 4. The first two columns hold document identifiers (document name and part); the following two hold sentence and token indexes, followed by the surface form of the word. Columns 6 through 9 hold predicted lemma, part-of-speech tag, morphological analysis, and named entity, respectively. The next three columns correspond to the syntactic structure: with the token index of the head word and the edge label according to the dependency tree (columns 10 and 11), followed by the constituent structure (column 12). Columns 13 and 14 encode the prosodic features, as described in Section 2.2. i.e. the pitch accents, followed by the boundary tones. As outlined above, multiple pitch accents are concatenated. For instance, the second token in the first sentence (UNO-Sondergesandte) was realised with two pitch accents, a rising and a falling one, and was followed by an intermediate phrase boundary. If a word was realised without a pitch accent or without a boundary tone, the respective entry in the column is marked with the label "NONE". In the absence of an adequate mapping of the spoken realization and the textual tokenization, the label  Table 4: Column numbers, content our format "N/A" was applied in DIRNDL anaphora , for instance in the case of punctuation tokens or major deviations due to slips of the tongue. This label was also used for those cases, where no prosodic annotations were available. The final three columns represent the pragmatic annotations: first the lexical layer (column 15), then the referential layer (column 16). The very last column encodes coreference, by grouping mentions into sets with common identifiers, as is the case in the CoNLL shared task format (Pradhan et al., 2012). For instance, the mention des Kosovo is R-UNUSED and belongs to the coreference cluster with id 901. The word Kosovo as such is labeled L-NEW. In addition to the referential and lexical information status labels, each mention that bears such a label has a unique identifier associated, separated by the $ sign, i.e., des Kosovo has the identifier 6372 in the referential layer and Kosovo has the identifier 10513 in the lexical layer. The purpose of these identifiers is to simplify parsing the format in case of nested mentions, e.g. the full phrase für eine Unabhängigkeit des Kosovo unter internationaler Aufsicht is labeled R-BRIDGING-CONTAINED, 5 whereas the underlined subphrase is labeled R-GENERIC. Since some of the RefLex labels have anchors in other mentions, these are also included as part of the labels in columns 15 and 16, separated by another $. For instance, the L-GIVEN-SUPER on Provinz in the second sentence indicates that this word is a hypernym of its anchor. The anchor is indicated by the last part of the label in column 15, 1-9-9, which denote sentence number, first token, and last token, respectively. That is, the anchor for this label is the span Kosovo in the first sentence. As mentioned above, in addition to the format described here, the release of the corpus also includes two other tabular versions of the same data. Specifically, it includes the tables in the CoNLL 2011/2012 shared task format as well as the SemEval 2012 shared task format.

Conclusion
We presented DIRNDL anaphora , a resource for anaphora resolution created from the extended DIRNDL corpus, which contains spoken and written radio news amounting to roughly 50,000 words. The corpus has been manually annotated for prosodic and pragmatic information. The new version includes a revised and updated version of the pragmatic annotations, as well as automatic predictions by state-of-the-art morphosyntactic tools, including part-ofspeech and morphology as well as dependency and phrasestructure syntactic trees. Since our explicit goal is to enable developers of automatic tools for coreference and bridging resolution to use DIRNDL anaphora as a resource for evaluation, we are making the corpus available for download 6 in established textbased formats previously used for coreference resolution.