{'fraction_non_alphanumeric': 0.032969456389899655, 'fraction_numerical': 0.013121623111699196, 'mean_word_length': 4.978576137112722, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The BDCamões Collection of Portuguese Literary Documents: a Research Resource for Digital Humanities and Language Technology

This paper presents the BDCamões Collection of Portuguese Literary Documents, a new corpus of literary texts written in Portuguese that in its inaugural version includes close to 4 million words from over 200 complete documents from 83 authors in 14 genres, covering a time span from the 16th to the 21st century, and adhering to different orthographic conventions. Many of the texts in the corpus have also been automatically parsed with state-of-the-art language processing tools, forming the BDCamões Treebank subcorpus. This set of characteristics makes of BDCamões an invaluable resource for research in language technology (e.g. authorship detection, genre classification, etc.) and in language science and digital humanities (e.g. comparative literature, diachronic linguistics, etc.).


Introduction
This paper introduces the BDCamões Collection of Portuguese Literary Documents, 1 a new language resource that is suitable for the scientific study and the technological preparation of the Portuguese language. This corpus is a collection of written documents with close to 4 million words from over 200 documents. In contrast to many existing corpora aimed at primarily supporting the development and testing of natural language processing tools and applications, BDCamões has a number of less common, very interesting set of characteristics that together turn it into an invaluable research resource, complementing other related resources. It is made of complete documents, rather than of fragments; it is made of high quality, carefully edited texts, rather than of content that has been (semi-)automatically scrapped from web pages; it contains texts from a wide time span ranging from the 16th century to the 21st century, rather than only from recent times, or only from old times; it is made mostly of literary texts, rather than from the more usual, easily available domains of news articles, legal documents, etc.; it includes texts of different genres, such as novels, chronicles, poems and short stories, among others; it contains texts from a number of different authors, in different styles, rather than originating from a single author or adhering to a uniform style; its documents have well identified authors, rather than lacking clear authorship; many of its texts are outstanding landmarks of culture expressed in Portuguese language and/or are of high historical relevance (e.g. the first theater plays written in Portuguese) or from the greatest authors (e.g. Luís de Camões, Eça de Queirós, Fernando Pessoa, etc.); and last but not least, its texts adhere to a range of different orthographic traditions or standards that have been used for the Portuguese language. The BDCamões corpus is distributed by PORTULAN CLARIN, the Research Infrastructure for the Science and Technology of Language, 2 under the most permissive licenses possible given their different intellectual property rights. Given its unique set of features, this corpus turns out to be a highly versatile language resource suitable for numerous research purposes for the science and technology of the Portuguese language. This is strengthened by the circumstance that, alongside the raw text versions, it includes also versions of many of its documents where these were automatically annotated and parsed with a wide range of linguistic information, including on part-of-speech, morphological features, grammatical dependencies and on expressions denoting named entities. Focusing on language technology, the BDCamões corpus can be used to support the development of computational processing tools for authorship detection, genre classification, grammar checking, orthographic conversion, lexicon construction, etc., on a par of course with the more usual processing tools whose development is also supported by other types of corpora. Focusing on language science, in turn, this corpus offers a great potential for the research in digital humanities. It will make viable the study of literary works and authors enhanced by computational technology solutions and thus under a new light that previous methods would not support. For instance, it allows for: the rapid development of (sub-)vocabularies; accurate indexes of words and their occurrence in the context of specific works or authors; comparative studies on different literary schools, different authors or different creative periods of a given author; diachronic studies concerned with the evolution of the Portuguese language; among many others. This paper is organized as follows. The next Section 2 provides an overview of some of the existing corpora for Portuguese that may be closer to BDCamões and in what way it differs from them. Section 3 describes in detail the content of the corpus and how it was gathered, while Section 4 informs about its distribution and licenses. Section 5 closes the paper with concluding remarks.

Related work
There are a few corpora for Portuguese that are suitable to support language research and the development of language technology. The BDCamões corpus complements them and opens new possibilities for research and innovation that were not so amply available due to its unique set of characteristics. We put it in contrast to some more relevant language resources with which it can be more closely compared with. CIPM-Corpus Informatizado do Português Medieval (Xavier, 2016) is a corpus of 2,670 texts, totaling 2 million words, from the 12th to the 16th century, comprising several genres, such as historical narratives, religious texts and poetry. It addresses an earlier time span not in BDCamões but lacks coverage from the 16th century onward. CTA-Corpus de Textos Antigos contains 29 historiographic texts as well as hagiographic, spiritual and novelistic texts originally written or translated into Portuguese until 1525. 3 Tycho Brahe-Parsed Corpus of Historical Portuguese (Galves, 2018) is a corpus of texts written in Portuguese between the 15th and 19th centuries, with 76 texts from over 50 authors, comprising 3.3 million tokens, which only partly coincide with the texts in BDCamões (6 texts, with about 159,000 words). LT Corpus-Corpus de Textos Literários (Généreux et al., 2012) is a literary corpus containing 70 documents published between the mid-19th century and the 1940's of the 20th century. While similar in design to and complementing BDCamões, it covers a shorter time span, has less variety of genres, fewer authors, and is smaller in size, with 1.8 million words, which only partly coincide with the texts in BDCamões (23 texts, with about 897,000 words). CINTIL-Corpus Internacional do Português (Barreto et al., 2006) is a linguistically interpreted corpus with 1 million tokens, mostly from anonymized excerpts of news articles, but also including some works of fiction, and transcriptions of formal and informal speech. It is annotated with a variety of manually verified linguistic information, including morphological information and part-of-speech tags. Its texts are only from a recent period and it lacks some metadata items, such as information on the author, that would be necessary for some types of studies.

Documents gathering methodology
The books to which the documents in the BDCamões corpus relate to were collected and converted to their digital versions in PDF format under appropriate licensing in a dedicated acquisition and processing campaign in Camões I.P., the Portuguese official national organization, which acts under indirect administration of the Portuguese Ministry of Foreign Affairs, that is responsible for proposing and implementing Portuguese language and culture internationalization policies. These documents, in their digital versions, were deposited in the Digital Library of Camões 4 -which gives the name to the present corpus -, from where they can be freely retrieved and used under the respective licensing conditions. The documents were also further processed to produce the BDCamões corpus, which was deposited in PORTULAN CLARIN Research Infrastructure for the Science and Technology of Language, 5 belonging to the Portuguese National Roadmap of Research Infrastructures of Strategic Relevance, and part of the international research infrastructure CLARIN ERIC. The corpus can be freely retrieved from its repository and used under the licensing conditions indicated below in Section 4. The PDF files were either produced from digital scans of the pages of the corresponding physical documents or exported to that format by the editors who allowed them to be published in BDCamões. While these files represent the visual aspect of the original documents, they cannot be processed as text by language processing tools. The PDF documents were converted in the University of Lisbon into files in plain text format using the command line tool PDFTOTEXT, 6 which extracts any textual content found within a PDF file. This extraction was feasible because, when the PDF files were obtained from the scans of hard books, they underwent a process of optical character recognition (OCR) that secured a textual version of the content within the file. The texts extracted from PDF files contained many errors, for instance, mistaking l (lowercase "L") for I (uppercase "i"), or rn for m, etc., which are indicative of failures in the preceding optical character recognition (OCR) process used to originate the PDF files. Since there is no safe heuristic to automatically detect and fix such cases, we performed an exhaustive manual revision of the converted plain text documents and the errors were corrected by two linguists, the first two authors of this paper, taking into account the input PDF version of the documents. Note that the manual correction only addressed the errors introduced by the conversion. The texts were otherwise transcribed literally, including eventual typographic errors present in the original edition. The conversion to plain text is necessarily lossy in what 4 https://www.instituto-camoes.pt/ en/activity-camoes/online-services/ service-desk 5 https://portulanclarin.net 6 The PDFTOTEXT tool is part of the XPDF toolkit (http:// www.xpdfreader.com).
concerns some aspects of formatting (e.g. font style) and layout (e.g. headers and footers). For BDCamões, page headers and page numbering were removed, while the tables of contents (if applicable) and footnote content were preserved. The content of a footnote is placed at the next available paragraph break after its reference so as not to break the sentence where the footnote is referred to. The construction of the corpus is an ongoing work, and the texts included in this inaugural version are those whose conversion to digital version and subsequent curation has been already concluded.

Corpus statistics
The BDCamões corpus is composed of 208 documents and has a total of 3,945,943 words. 7 There are 83 authors whose texts are included in the corpus (see Table 1 for the full list of authors  , in turn, is the author with the largest volume of texts, in terms of word count, with over 13% of the words in the corpus coming from his 5 works (4 novels and 1 tale). Here we consider "word" as any sequence of characters delimited by white space, and the count is obtained by the standard Linux command line tool WC. The corpus covers written texts from several genres, such as tales, novels, chronicles, poems, dramas and essays, among others, as shown in greater detail in Table 2. Similarly to what happens regarding authorship, the proportion of documents and words for each genre varies. Tales are the most common genre in number of documents, providing for more than 44% of the texts in the corpus. As a result of their smaller size, they only account for 17% of the corpus in terms of words. The much longer novels, though being only 12% of the documents, account for over 32% of the words in the corpus. In terms of the time span represented, the corpus contains texts from the 16th century onward. Namely, 7 from the 16th century, 4 from the 17th century 8 from the 18th century, 84 from the 19th century, 82 from the 20th century and 23 from the 21st century. As such, this corpus represents different phases of the Portuguese language, including 13 texts from Middle Portuguese (up to early 16th century) or Classic Portuguese (until mid-18th century). The remaining text are in some form of Modern Portuguese (from the mid-18th century onward; or older but in an edition that has been transcribed into those orthographic norms): 21 are written according to the Portuguese orthographic norm of 1911, and 174 according to the norm of 1945. The authors, genres and time periods are not equally represented in the collection, as the goal of the effort described here is to gather and transcribe the documents available in <document> <header> <title> ... </title> <author> ... </author> <type> ... </type> <firstPublicationDate> ... </firstPublicationDate> <publisher> ... </publisher> <publicationDate> ... </publicationDate> </header> <text> ... </text> <annotation> ... </annotation> </document> Figure 1: XML structure of a document in BDCamões the Digital Library of Camões, making them available for various types of studies. Researchers interested in a particular set of authors, genre or time period will then be able to take the BDCamões corpus as a resource in which the relevant documents may be found.

Metadata and linguistic annotation
All the documents written in Modern Portuguese, or which are older but whose edition has been transcribed into that orthographic norm, have been automatically parsed with state-of-the-art language processing tools for Portuguese , and thus annotated with linguistic information that follows from the design of these tools and that can be found in detail in their guidelines and documentation (Branco et al., 2015).
This subcorpus forms the BDCamões Treebank with 4,495,379 tokens, of which 3,456,396 belong to the public domain part and 1,038,983 belong to restricted part of the corpus.
The resulting linguistic annotation comprises part-ofspeech tags (e.g. PREP, ADV, etc.), morphology (lemmas for words from the open categories; gender and number for words from nominal categories; tense, aspect, person and number for verbs), named entities (in BIO notation), syntactic analysis in terms of graphs of grammatical dependencies (e.g. SJ, OBL, M, etc.), and semantic analysis in terms of semantic roles (e.g. ARG1, ARG2, LOC, etc.). A second version of the dependency graphs was obtained by converting them to the so called Universal Dependencies (de Marneffe et al., 2014).
Each document is stored in a separate file, associated with the metadata record in XML markup shown in Figure 1. The text itself and, when applicable, the corresponding linguistically annotated data appear, respectively, in the two fields <text> and <annotation>. The remaining fields, in the header, contain the title, author and type (genre) of the work, and information on its publication (the date for the first publication of the work, and the publisher and data of the publication for the edition that was transcribed).
The output of the annotation uses a CoNLL-style format, with one token per line and tab-separated fields. An excerpt of an annotated sentence may be seen in Figure 2. The Universal Dependencies version of the grammatical dependencies appear in a ante-penultimate column, parallel to and after the column with the original analysis.  (2) normalized word form (e.g. after expanding contracted forms); (3) lemma; (4) part-of-speech; (5) inflection; (6) named entity (BIO notation); (7-8) dependency relation and parent index; (9-10) dependency relation and parent index, in Universal Dependencies; and (11) spacing around the token (e.g. LR indicates the token had spaces to the left and to the right of it in the original sentence).

Corpus distribution
The BDCamões corpus is distributed by PORTULAN CLARIN, the Research Infrastruture for the Science and Technology of Language. 8 The distribution is split into two parts, whose usage is ruled by different licensing conditions, namely Part I includes the documents that are in the public domain and Part II includes the remaining documents (cf. Table 3). The annotated subcorpus BDCamões Treebank is part of the distribution of the BDCamões corpus, and additionally, for the sake of the convenience of its users, it is also distributed separately, split into two parts as well, by PORTULAN CLARIN. The two parts of the BDCamões corpus are distributed separately, each having a different entry in the PORTULAN CLARIN repository. The two parts of BDCamões Treebank also have their own entries. The two parts of the corpus are distributed under the most permissive license for each of them. Part I of BDCamões is distributed under the license CC-BY, which requires that when used, the academic authorship of this part of the corpus is acknowledged. Part II has the license CC-BY-NC-ND, which restricts it to research, non commercial usage and does not allow its redistribution. The two parts of the treebank receive similar licenses.

Conclusion
In this paper we presented the inaugural version of the BDCamões corpus, a new language resource to support the science and technology of language. It is a collection of 208 complete documents, mostly literary, from 14 genres, by 83 authors, whose first publication dates cover a time span from the 16th to the 21th century, and totaling close to 4 million words. Its unique set of characteristics, namely 8 http://portulanclarin.net being composed of complete documents instead of fragments, the carefully edited text, the variety of genres and authors, and the wide time span and orthographic conventions covered, make of it an invaluable and versatile resource for multiple research purposes, both for language technology and digital humanities.
In future work, we aim at publishing further versions of the BDCamões corpus, extending it with further documents, and from further variants of Portuguese.