{'fraction_non_alphanumeric': 0.03346184622099367, 'fraction_numerical': 0.007710773259620281, 'mean_word_length': 4.832838353839627, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Statistical Method of Recognizing Local Cohesion

This paper presents a method for automatically recognizing local cohesion between utterances, which is one of the discourse structures in task-oriented spoken dialogues. More specifically we can automatically acquire discourse knowledge from an annotated corpus with local cohesion. In this paper we focus on speech act type-based local cohesion. The presented method consists of two steps 1) identifying the speech act expressions in an utterance and 2) calculating the plausibility of local cohesion between the speech act expressions by using the dialogue corpus annotated with local cohesion. We present two methods of interpolating the plausibility of local cohesion based on surface information on utterances. The presented method has obtained a 93% accuracy for closed data and a 78% accuracy for open data in recognizing a pair of utterances with local cohesion.


Introduction
For ambiguity resolution, processing of a discourse structure is one of the important processes in Natural Language Processing (NLP). Indeed, discourse structures play a useful role in speech recognition, which is an application of NLP. In the case of Japanese, it is very difficult to recognize the end in utterances by using current speech recognition techniques because the sound power of an ending tends to be small. For example, "desu", which represents the speech act type "response", is often misrecognized as "desu-ka (question)" or "desu-ne (confirmation)". On the other hand, Japanese can easily select the adequate expression "desu", when the intention of the previous utterance is concerned with a question. This is because they use the coherence relation (local cohesion) between the two utterances, question-response.
In the conventional approach (i.e., rulebased approach) to processing the discourse structure [Hauptmann 88][Kudo 90] [Yamaoka 91][Young 91], NLP engineers built discourse knowledge by hand-coding. However, the rulebased approach has a bottleneck in that it is a hard job to add discourse knowledge when the employed NLP system deals with a larger domain and more vocabulary.
Recently, statistical approaches have been attracting attention for their ability to acquire linguistic knowledge from a corpus. Compared with the above rule-based approach, a statistical approach is easy to apply to larger domains since the linguistic knowledge can be automatically extracted from the corpora concerned with the domain. However, little research has been reported in discourse processing [Nagata 94][Reithinger 95], while in the areas of morphological analysis and syntactic analysis, many research studies have been proposed in recent years. This paper presents a method for automatically recognizing local cohesion between utterances, which is one of the discourse structures in task-oriented spoken dialogues. We can automatically acquire discourse knowledge from an annotated corpus with local cohesion. In this paper we focus on speech act type-based local cohesion. The presented method consists of two steps ~1) identifying the speech act expressions in an utterance and 2) calculating the plausibility of local cohesion between the speech act expressions by using the dialogue corpus annotated with local cohesion. We present two methods of interpolating the plausibility of local cohesion based on surface information on utterances. Our method has obtained a 93% accuracy for closed data and a 78% accuracy for open data in recognizing a pair of utterances with local cohesion.
In Section 2, local cohesion in task-oriented dialogues is described. In Section 3, our statistical method is presented. In Section 4, the results of a series of experiments using our method are described.

Local Cohesion between Utterances
The discourse structure in task-oriented dialogues has two types of cohesion: global cohesion and local cohesion. Global cohesion is a top-down structured context and is based on a hierarchy of topics led by domain (e.g., hotel reservation or flight cancellation). Using this cohesion, a taskoriented dialogue is segmented into several subdialogues according to the topic. On the other hand, local cohesion is a bottom-up structured context and a coherence relation between utterances, such as question-response or response-confirmation. Different from global cohesion, local cohesion does not have a hierarchy. This paper focuses on local cohesion. Figure 1 shows a Japanese conversation between a person and a hotel staff member, which is an example of a task-oriented dialogue; The person is making a hotel reservation. The first column represents global cohesion and the second column represents local cohesion. For example, the pair of U3 and U4 has local cohesion, because it has a coherence relation for each word in the utterances as follows: c 1) speech act pattern between "onegaiitashimasu (requirement)"in U3 and "desu (response)" in U4 c2) semantic coherence between nouns, "hinichi (date)"in U3, and "hachi-gatsu to-ka (August 10th)" and "ju-ni-nichi (12th)"in U4 In the same way, (U4, U5) and (U5, U6) have local cohesion. Thus, U3 to U6 are built up as one structure and form a subdialogue with the topic "date". As observed from this example, whether two utterances have local cohesion with one another or not is determined by coherence relations between the speech act types in the utterances, coherence relations between the verbs in them and coherence relations between the nouns in them. In recognizing local cohesion, our method uses these three coherence relations.

Our Approach 3.1 Utterance Model with Local Cohesion
In this paper, we approximate an utterance in a dialogue to a three-tuple; U = (SPEECH __ ACT, VERB, NOUNS) (1) where SPEECH ACT is the speech act type, VERB is the ma]-n verb in the utterance, and NOUNS is a set of nouns in the utterance' (e.g., a subject noun and an object noun for the main verb). Figure 2 shows a dialogue with our utterance model.
nonnegatlve weights contributing to local cohesion, and cohesion_speechact, cohesion_verb and cohesion noun are functions giving the plausibility of coherence relations between speech act types, verbs and nouns respectively. The problem of deciding an utterance that has local cohesion with U. can then be formally defined as finding a J utterance with the highest plausibility of local cohesion for Uj, which is the result of the following function:

Uopt s = arg max0<i<]_l cohesion(U i , U] )
(3) As the first step, this paper uses only speech act types in the calculation (i.e. A t =1, J 2 = J 3 = 0). This is because the speech act types are more powerful in finding local cohesion than the verbs or the nouns as follows: rl) The speech act types are independent of domain.
r2) The speech act types are stable, while the nouns and the verbs are sometimes omitted in utterances in spoken dialogues. Thus, Equation (2) is reduced to: In order to calculate Equation (4), two kinds of information to answer the following questions are required as discourse knowledge: q l) What expressions in an utterance indicates a speech act type? q2) What speech act pattems have local cohesion? We automatically acquire these discourse knowledge from a annotated corpus with local cohesion. According to the information, our method is composed of two processes, 1) identifying the expressions which indicate a speech act type (called speech act expressions) in an utterance and 2) calculating the plausibility of the speech act patterns by using the dialogue corpus annotated with local cohesion.

Identifying Speech Act Expressions in an Utterance
The first process in our method identifies the speech act expression in each of the utterances by matching the longest pattern with the words in a set of speech act expressions. The words can be collected by automatically extracting fixed expressions from the parts at the end of utterances, because the speech act expressions in Japanese have two features as follows: fl) The speech act expression forms fixed patterns.
f2) The speech act expressions lie on the parts at the end of the utterance. For example, "desu" in "Futari desu"(U8) in Figure 1 represents a speech act type "response". We call these expressions ENDEXPR expressions. For the automatic extraction, we use a slight modification of the cost criteria-based method [Kita 94], which uses the product of frequency and length of expressions, because this is easy when dealing with languages that do not use delimiters in their writings such as Japanese. Kita et al. extract expressions in the order of larger cost criteria values. We do so in the order of longer fixed-expressions with cost-criteria values above a certain threshold. For the more details of our extraction method, see [Katoh 95].
When the ENDEXPR expressions, which are listed in the set of the speech act expressions (represented as Set-ENDEXPR), are defined as a symbol ENDEXPR, we can approximate the speech act types as SPEECH ACT=ENDEXPR.

Calculating the Plausibility of Local Cohesion
The second process is to calculate the plausibility of local cohesion between utterances from the dialogue corpus using a statistical method. In this paper, we define the plausibility of local cohesion, i.e., Equation (5)

f (ENDEXPR,, ENDEXPRj ) = P(ENDEXPR~ ~ ENDEXPRj) P( ENDEXPRi n ENDEXPR j ) x log P( ENDEXPR i ) x P(ENDEXPRj )
Thefand f are modified functions of mutual information in Information Theory. We call them pseudo-mutual information. P(.) is the relative frequency of two utterances with local cohesion and P(.) is that of two utterances without local cohesion. ENDEXPR~ (~ ENDEXPR. means that ENDEXPR: appears next to ENDEXPRz. We call a series of two ENDEXPR expressions (i.e.,

ENDEXPR i 0 ENDEXPRi+I) an ENDEXPR bigram.
The htrger the value of Equation (6) in two utterances gets, the more plausible the local cohesion between them becomes. For example, in applications in speech recognition, the optimal result has the largest plausibility value among several candidates obtained from a module of speech pattern recognition.

Smoothing Methods
Although the statistical approach is easy to implement, it has a major problem, i.e., sparse data problem. Indeed Equation (6) gives very small values in some cases. In order to overcome this problem (to interpolate the plausibility), we propose two smoothing techniques.
Formally, when the speech act types are denoted by SACT TYPE, we can interpolate an original plausibilhy by using these type patterns:
The former method must use the n Xm parameters (i.e., ,u ~ ) and the latter one must produce the Iq speech act types. We chose the former method for our first experiments, because it was easier to implement.

Experiments
A series of experiments was done to evaluate our method. The experiments were carried out to decide whether one utterance and the next one have local cohesion with each other or not. The results of the experiment were able to segment a dialogue into several subdialogues.
First, Set-ENDEXPR was constructed automatically by our extraction method for fixed expressions from the ATR speech and language database [Morimoto 94]. The database includes about 600 task-oriented dialogues concerned with several domains, such as hotel reservation, flight cancellation and so on and each of the dialogues includes about 50 utterances on average. We have extracted about one hundred fixed-expressions from the database by the extraction method. Table  1 shows examples of the fixed expressions. Secondly, we chose 60 dialogues from the ATR database and annotated them with local cohesion by hand-code such as that shown in Figure 1 in Section 2. Then, six dialogues were taken from the 60 dialogues to use in testing for the open data, and the rest of the dialogues (54 dialogues) were used to calculate ENDEXPR bigrams. Moreover, six dialogues were taken from the 54 dialogues for the closed data. Using the 54 dialogues, the ENDEXPR bigrams were produced in the following four parts: An ENDEXPR bigram part A : with local cohesion + turn-taking. part B : with local cohesion + no turn-taking. part C : without local cohesion + turn-taking. part D : without local cohesion + no turn-taking. where "turn-taking" means that an utterance and the next one are produced by different persons and "no turn-taking" means that they are done by the same person.  Table 3 shows the accuracy of recognizing local cohesion in three cases: the turn-taking case, the no turn-taking case and the total case.  Table 3, the "default method" assumed that all of the pairs of utterances in a dialogue has local cohesion, and the accuracy was calculated as: The accuracy in the "default method" The number of the pairs of utterances . wltn local coneston = The total number of the pairs of utterances in a dialogue As shown in Table 3, the accuracy of our method was higher than that of the "default method". Using the smoothing method, we obtained a 93.8% accuracy for the closed data and about a 78.4% accuracy for the open data.

Conclusion
We described our statistical method of recognizing local cohesion between utterances, based on pseudo-mutual information. We focused on speech act expressions in calculating the plausibility of local cohesion between two utterances. The results of the first experiments showed a 93% accuracy for closed data and a 78% accuracy for open data. We conclude that the presented method will be effective for recognizing local cohesion between two utterances. To improve our method, we will use the coherence relations in the verbs and set of nouns and will use a larger corpus with local cohesion. We plan to apply the method to speech recognition in a speech-to-speech machine translation system.