{'fraction_non_alphanumeric': 0.03552365175777655, 'fraction_numerical': 0.007546475243879993, 'mean_word_length': 5.091087443946188, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering

This paper presents the QALL-ME benchmark, a multilingual resource of annotated spoken requests in the tourism domain, freely available for research purposes. The languages currently involved in the project are Italian, English, Spanish and German. It introduces a semantic annotation scheme for spoken information access requests, specifically derived from Question Answering (QA) research. In addition to pragmatic and semantic annotations, we propose three QA-based annotation levels: the Expected Answer Type, the Expected Answer Quantifier and the Question Topical Target of a request, to fully capture the content of a request and extract the sought-after information. The QALL-ME benchmark is developed under the EU-FP6 QALL-ME project which aims at the realization of a shared and distributed infrastructure for Question Answering (QA) systems on mobile devices (e.g. mobile phones). Questions are formulated by the users in free natural language input, and the system returns the actual sequence of words which constitutes the answer from a collection of information sources (e.g. documents, databases). Within this framework, the benchmark has the twofold purpose of training machine learning based applications for QA, and testing their actual performance with a rapid turnaround in controlled laboratory setting.


Introduction
This paper presents the QALL-ME benchmark, a multilingual resource of annotated spoken requests in the tourism domain, freely available for research purposes. The QALL-ME benchmark is developed under the EU-FP6 QALL-ME project 1 which aims at the realization of a shared and distributed infrastructure for Question Answering (QA) systems on mobile devices (e.g. mobile phones). Within this framework, our main motivation is to deliver a humanannotated resource for QA systems development and evaluation. Since the project deals with both textual and spoken requests, the annotation of the resource pays particular attention to the information needed in the QA and speech processing research areas. Annotation levels include both pragmatic and semantic annotations. Moreover, additional layers specifically referring to QA processing have been considered to fully capture the relevant information for more general applications. In particular, we introduced three QA-based annotation levels: Expected Answer Type, Expected Answer Quantifier, and Question Topical Target. To the best of our knowledge, none of the currently available annotated corpora of spoken language dealing with information requests contains QA specific labels. Therefore, our contribution aims at improving the proposed annotation schemes by considering specific information broadly and successfully exploited in QA. The benchmark currently includes 14645 questions in four different languages (Italian, Spanish, English, and German), related to the domain of cultural events in a town (e.g. cinema, theatre, exhibitions, etc.). The paper is structured as follows: Section 2 provides an overview of the QALL-ME benchmark as inserted in the project frame; Section 3 presents the data collection, with respect to the spoken language acquisition, the transcription and the translations of the data into English; Section 4 introduces the annotation layers, in particular the annotation of the speech acts; Section 5 presents the three QA-based annotation levels: the Expected Answer Type, the Expected Answer Quantifier and the Question Topical Target; Section 6 describes the annotation of the relations contained in the questions; Section 7 concerns related work; finally, Section 8 discusses future work and draws some conclusions.

The QALL-ME Benchmark
This Section shortly introduces the QALL-ME benchmark under the perspective of the QALL-ME project. The general objective of the project is to establish a shared infrastructure for multilingual and multimodal open domain Question Answering for mobile phones. The scientific and technological objectives pursue three crucial directions: multilingual open domain QA, user-driven and context-aware QA, and Machine Learning technologies for QA. The specific research objectives of the project include stateof-the art advancements in the complexity of the questions handled by the system (e.g. how questions); the development of a web-based architecture for cross-language QA (i.e. question in one language, answer in a different language); the realization of real time QA systems for concrete applications; the integration of the temporal and spatial context both for question interpretation and for answer extraction; the development of a robust framework for applying minimally supervised machine learning algorithms to QA tasks; and the integration of mature technologies for automatic speech recognition within the open domain question answering framework. The selected domain is represented by local events in a town, usually available either through specialized web sites or local newspapers and publications. Experimentations will be carried out in four cities (one for each language involved in the project), using constantly updated information provided by a number of selected data providers. In the project context, we have been developing two strategic resources: the QALL-ME Ontology, a formal representation of the domain of cultural events, and the QALL-ME benchmark, a corpus of multilingual annotated questions. The two resources are strictly connected as far as semantic annotations are concerned, as the Ontology provides semantic labels for the annotation of Expected Answer Type, Question Topical Target and questions relations. The use of the QALL-ME benchmark as a training for machine learning based algorithms for question interpretation is reported in (Negri et al., 2008). Both the QALL-ME benchmark and the QALL-ME ontology are being made incrementally available at the Project website (http://qallme.fbk.eu), where new updated versions in any of the four languages are published once a new annotation layer is completed.

Spoken Requests Acquisition
For data acquisition, a large number of speakers has been presented with a graphical interface, describing possible information needs in the selected domain. For each scenario two utterances were collected: the first one is spontaneous, while the second one is a pre-defined question that is simply read by the speaker. In order to minimize the risk of influencing the speaker in the formulation of the spontaneous utterances, each scenario was presented to them on a computer screen as a list containing: the context in which the question has to be posed (e.g. "Cinema/Movie" or "Concert"); the type of information the speaker wants to obtain from the system (e.g. the telephone number of a cinema, the cost of a ticket); a list of items that must be present in the question in order to ensure its validity (e.g. the name of the cinema is "Astra", the title of the opera is "La Boheme"); a list of additional items that the speaker can use to make the question (e.g.the cinema is located in "Via Manci", the concert venue is "Teatro Sociale"). Each question was acquired using a telephone, and recorded together with information for identifying the corresponding scenario.

Transcription
After the acquisition, all the audio files acquired from a speaker were joined together and orthographically transcribed using the tool Transcriber 2 . For each session, a dedicated transcription file was initialized, which includes time markers, the text of the read sentences, and the gender and accent of the speaker. Being domain-restricted, our scenarios often led to the same utterance (matching word sequence). However, the number of repetitions is actually small and concentrated within the read utterances; the repetitions are well documented in the resource, where the repeated utterances have been clustered. The number of distinct utterances, i.e. non repeated ones is: 3289 for Italian, 2427 for Spanish, 3472 for English and 796 for German. Data concerning the total speech duration, and the distribution of the speakers with respect to their language, gender and mother tongue is reported in Table 1 34 . Data concerning the resulting database are reported in Table 2.

Translations
The collected data have been translated into English by simulating the real situation of an English speaker visiting a foreign city, i.e. with non-translated named entities (e.g. names of streets, restaurants, etc.). One of the future goals is to have all the data collected for one language translated into the other three languages (using English as an interlingua, if necessary). The study on the portability of annotation layers from one language to another is in the pipeline.

Speech Acts Annotation
Besides the translation of the collected data into English, the QALL-ME benchmark addresses two main levels of annotation. The first one refers to speech acts, while the second introduces relevant elements for the semantic interpretation of the request, including Question Topical Target, Expected Answer Type and Expected Answer Quantifier. Transcribed files were annotated using CLaRK, an XMLbased System for Corpora Development 5 . On the speech act side, we separate within each utterance what has to be interpreted as the actual request from what does not need an answer. Request labels identify all the utterances used by the speaker to ask for information. Requests are marked either as DIRECT or INDIRECT. DI-RECT requests include wh-questions (as shown in Example 1), questions introduced by e.g. "Could you tell me", or "May I know", or pronounced with an ascending intonation (typical of Italian spoken questions). On an intuitive level, we can say that a request is DIRECT if we can put a question mark at its end (punctuation is actually not present in our corpus). Conversely, INDIRECT requests include requests formulated in indirect or in implicit ways, as shown  in Example 2. For non-request acts (utterances used by the speaker to introduce or contextualize the request), we use the label GREETINGS, THANKS, ASSERT (usually referred to as "declarative clause" as in (Soria and Pirrelli, 1999)), and OTHER, which includes non request utterances such as "well", "hallo", and "listen". To date, this level of annotation has been completed only for Italian (see (Cabrio et al., 2007)) and Spanish. The inter-annotator agreement has been calculated for Italian using the Dice coefficient, over 1000 randomly picked sentences annotated by two annotators. The Dice coefficient is computed as 2C/(A+B), where C is the number of common annotations, while A and B are respectively the number of annotations provided by the first and the second annotator. The overall agreement is 96.1%, with the following label breakdown: ASSERT:

Speech Acts Annotation for the English section
For speech acts annotation on the English section of the QALL-ME benchmark, a slightly different scheme is applied using the multipurpose annotation tool PALinkA (Orasan, 2003). Labels such as GREETING, THANKING, THANK-BYE and REQUEST-INFO are adapted from existing speech acts theories and dialogue annotation projects (see, e.g. (Larsson, 1998) for an overview/comparison) to suit our data. First, utterances are marked as suitable (<utterance>) or unsuitable (<interrupted>, <trash>, <nonsense>); then, C-units (Biber et al., 1999) are marked within suitable utterances. The C-UNIT tag takes the attributes CLAUSAL and NON-CLAUSAL; NON-CLAUSAL is further split into PRAGMATIC and SPECIFY-INFO. Next, the speech acts themselves are annotated. There are two general tags, PRIMARY SPEECH ACT and SECONDARY SPEECH ACT, the attributes assigned to which determine the attribute given to the final tag, SPEECH ACT TYPE, which is marked as a relation between the two Speech Acts tags. PRIMARY  Example 3: Speech acts .

Question Answering Annotation
This section describes the QA-based annotation levels, in particular the Expected Answer Type, the Expected Answer Quantifier, and the Question Topical Target.

Expected Answer Type (EAT).
The EAT has been defined by (Prager, 2007) as the class of object (or rhetorical type of sentence) sought by the question; in other words, it is the semantic category associated with the desired answer, chosen from a predefined set of labels. For EAT annotation, we extracted our EAT taxonomy from Graesser's taxonomy (Graesser et al., 1988), adding two other levels: one is based on the QALL-ME ontology, a domain specific ontology developed specifically for the project purposes; the other one is based on Sekine's Named Entity Hierarchy (ENE). 6 In detail, the level related to Graesser's taxonomy is domain independent and includes labels as FACTOID, PROCEDURAL, VERIFICATION, and DEFINITION/DESCRIPTION. Deeper levels (referring only to FACTOID questions tend to be more domain dependent, e.g. FACTOID EATs take semantic labels such as PER-SON, LOCATION, ORGANIZATION, and TIME, referring to the QALL-ME ontology (see Example 4). Concerning VERIFICATION, the definition of the question type is not enough, since the speaker implicitly needs more information than simply a yes/no answer (e.g. "is there a web-site of the police headquarters?"). These questions have thus been annotated both with the tag VERI-FICATION and the appropriate tags of the QALL-ME ontology (e.g. CONTACT). The choice of the correct EAT is not always straightforward and it is difficult to define unambiguous guidelines. For PROCEDURAL, and DEFINI-TION/DESCRIPTION types, no deeper levels have been defined.
To enhance a broad use of the benchmark also for opendomain QA applications, we annotated the EAT also based on Sekine's ENE, a shared EAT taxonomy. For the annotation task we used Sekine's tagging tool FuuTag.

Expected Answer Quantifier (EAQ).
We define the EAQ as an attribute of the EAT that specifies the number of expected items in the answer. Even though EAQ identification is usually not explicitly addressed in QA systems, the rationale behind this attribute has been implicitly asserted in the framework of the TREC and CLEF QA tasks, where test questions asking for multiple answer items are marked as "list" questions. For EAQ annotation, the possible values are: one, at least one, all, n.

Question Topical Target (QTT).
The QTT (sometimes referred to as question focus (Monz, 2003), or question topic (Prager, 2007)) is the part of text, within the question, that describes the entity about which the request has been made. We define the extension of the QTT as the whole syntactic phrase (noun or verb phrase) whose head is the entity about which something is asked, as in: "How much does it cost to get to Santa Chiara hospital by taxi?"(QTT is underlined). Especially in the Document Retrieval phase of the QA process, QTT identification becomes useful: since QTT terms (or their synonyms) are likely to appear in a retrieved sentence that contains the answer, query formulation/relaxation strategies should appropriately weight such terms (Monz, 2003). However, especially when dealing with complex queries, more than one candidate QTT can be found, and their identification is not always straightforward. Since more than one QTT may appear in the same utterance, we introduced a QTT identifier to allow for EAT references, as shown in Example 5. While an EAT always refers to a single QTT, a QTT can have one or more associated and possibly different EATs (e.g. when asking for both time and place of an event).

Annotation of Relations
In order to enhance a richer semantic interpretation of the questions, also the annotation of the relations that they contain has been addressed. Such annotation is work in progress, and will be completed in future releases of the QALL-ME benchmark. Detecting relations among entities is often crucial, especially in QA applications, as they convey and complete the context in which a specific request has to be interpreted. Often, in fact, discovering relations is necessary to capture all the constraints that define the actual information need expressed by the request, thus defining and narrowing the search space of potential answers. For instance, the relations between a MOVIE and the DATE of its projection, the MOVIE and the STARTINGHOUR of a specific show, and a MOVIE and the CINEMA where it is projected must be taken into account while interpreting the question: "at what time is the movie il grande capo beginning tomorrow afternoon at Vittoria cinema". At this stage the annotation focuses only on binary relations. For this purpose, a total of 75 relations defined in the QALL-ME ontology have been selected. • "when will Eragon be on in Trento" • "what is the name of the director of dreamgirls today at Nuovo Roma cinema" • "which dramatic movie directed by Gabriele Muccino is now showed" As can be seen from the previous examples (specifically the first and the second) relation annotation is related to EAT annotation, with partial overlaps. Often, in fact, the EAT of a question (e.g. TIME) can be mapped to the range of one of the annotated relations (e.g. STARTINGHOUR).
In the current version of the QALL-ME benchmark around 10% of the Italian questions (367 out of 3289), namely those referring to the Cinema/Movie domain, have been annotated with the 12 (out of 75) relations that hold in such domain. Even though this is a relatively small subset of the whole benchmark, it's worth noting that all the relations annotated for a specific question are portable across languages, being our translations strictly literal. As

Related Work
In recent years, a number of research projects supported spoken dialogue annotation at different levels, with the purpose of creating language, domain, or task-specific benchmarks. Depending on the specific developers' purposes, the proposed annotation schemes cover a broad variety of information, ranging from the syntactic to the semantic and pragmatic level. Released in the nineties, the ATIS and TRAINS corpora 7 are collections of task-oriented dialogues in relatively simple domains. The former contains speech data related to air travel information, and is partially annotated (2,900 out of a total of 7,300 utterances) with reference answers, and a classification of sentences into i) those dependent on context for interpretation, ii) those whose interpretation does not depend on context, and iii) the not evaluable ones. The latter includes 98 dialogs (6,5 hours of speech, 55,000 transcribed words), dealing with routing and scheduling of freight trains. Utterances are annotated with dialogue acts (or "Communicative Functions") including, among others, the types INFO-REQUEST, EXCLAMATION, EXPLICIT-PERFORMATIVE, and ANSWER. More recently, the VERBMOBIL project (http://verbmobil.dfki.de) on speech-to-speech translation released large corpora (3,200 dialogs, 181 hours of speech, 1,520,000 running words) for German, English, and Japanese. Part of such material (around 1,500 dialogs) is annotated with different levels of information including: orthography, segmentation, prosody, morphology and POS tagging, semantic and dialog acts annotation. The latter annotation level has been carried out considering a hierarchy of 32 dialog acts such as GREET, THANK, POLITENESS FORMULA, and REQUEST. Spoken dialogue material collected within the MATE project 8 refers to any collection of spoken dialogue data (human-human, human-machine), including not only speech files, but also log-files or scenarios related to spoken dialogue situations. The annotation levels include prosody, morpho-syntax, co-reference, communication problems, and dialogue acts (e.g. OPENING, ASSERT, INFO REQUEST, ANSWER). Finally, the ongoing project LUNA (http://www.istluna.eu) is developing a multilingual and multidomain spoken language corpus, with the transcription and the semantic annotation of human-human and human-machine spoken dialogs collected for different application domains (call routing, travel information) and languages (French, Italian and Polish). At present, the completed annotation layers concern the argument structure, co-reference/anaphoric relations, and dialog acts. Even though the proposed annotation schemes proved to be suitable for specific information access systems, we believe that additional layers referring to QA processing should 7 http://www.ldc.upenn.edu/Catalog 8 http://mate.nis.sdu.dk audio transcr.
transl. speech acts EAT Sekine EAT ontol. ITALIAN X X X X X X SPANISH X X X X X May 08 ENGLISH X X -April 08 April 08 April 08 GERMAN June 08 June 08 undefined undefined undefined undefined Table 4: Present situation and tentative scheduling of the availability of the resource.
be considered to fully capture the relevant information for more general applications.

Conclusions and Future Work
This paper presented the QALL-ME benchmark, a multilingual resource (for Italian, Spanish, English and German) of annotated spoken requests in the tourism domain. The benchmark takes into account the importance of annotation layers specifically referring to the QA area. The present situation is summarized in Table 4. According to the QALL-ME project agenda, the above mentioned annotation layers will be completed, for all languages involved, during the second year of the project (due to technical problems, the scheduling of the availability of the resource for German is still undefined). Additional layers will be considered in the future: these include Multiwords, Named Entities, and normalized Temporal Expressions. The expected result is a reference resource, useful to train and test information access models not limited to QA.