{'fraction_non_alphanumeric': 0.03912716328066215, 'fraction_numerical': 0.012958782710475713, 'mean_word_length': 4.527726432532347, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Neighbors Help: Bilingual Unsupervised WSD Using Context

Word Sense Disambiguation (WSD) is one of the toughest problems in NLP, and in WSD, verb disambiguation has proved to be extremely difficult, because of high degree of polysemy, too fine grained senses, absence of deep verb hierarchy and low inter annotator agreement in verb sense annotation. Unsupervised WSD has received widespread attention, but has performed poorly, specially on verbs. Recently an unsupervised bilingual EM based algorithm has been proposed, which makes use only of the raw counts of the translations in comparable corpora (Marathi and Hindi). But the performance of this approach is poor on verbs with accuracy level at 25-38%. We suggest a modification to this mentioned formulation, using context and semantic relatedness of neighboring words. An improvement of 17% 35% in the accuracy of verb WSD is obtained compared to the existing EM based approach. On a general note, the work can be looked upon as contributing to the framework of unsupervised WSD through context aware expectation maximization.


Introduction
The importance of unsupervised approaches in WSD is well known, because they do not need sense tagged corpus. In multilingual unsupervised scenario, either comparable or parallel corpora have been used by past researchers for disambiguation (Dagan et al., 1991;Diab and Resnik, 2002;Kaji and Morimoto, 2002;Specia et al., 2005;Lefever and Hoste, 2010;Khapra et al., 2011). Recent work by Khapra et al., (2011) has shown that, in comparable corpora, sense distribution of a word in one language can be estimated using the raw counts of translations of the target words in the other language; such sense distributions contribute to the ranking of senses. Since translations can themselves be ambiguous, Expectation Maximization based formulation is used to determine the sense frequencies. Using this approach every instance of a word is tagged with the most probable sense according to the algorithm.
In the above formulation, no importance is given to the context. That would do, had the accuracy of disambiguation on verbs not been poor 25-35%. This motivated us to propose and investigate use of context in the formulation by Khapra et al. (2011).
For example consider the sentence in chemistry domain, "Keep the beaker on the flat table." In this sentence, the target word 'table' will be tagged as 'the tabular array' sense since it is dominant in the chemistry domain by their algorithm. But its actual sense is 'a piece of furniture' which can be captured only if context is taken into consideration. In our approach we tackle this problem by taking into account the words from the context of the target word. We use semantic relatedness between translations of the target word and those of its context words to determine its sense.
Verb disambiguation has proved to be extremely difficult (Jean, 2004), because of high degree of polysemy (Khapra et al., 2010), too fine grained senses, absence of deep verb hierarchy and low inter annotator agreement in verb sense annotation. On the other hand, verb disambiguation is very important for NLP applications like MT and IR. Our approach has shown significant improvement in verb accuracy as compared to Khapra's (2011) approach.
The roadmap of the paper is as follows. Section 2 presents related work. Section 3 covers the background work. Section 4 explains the modified EM formulation using context and semantic relatedness. Section 5 presents the experimental setup.
Results are presented in section 6. Section 7 covers phenomena study and error analysis. Conclusions and future work are given in the last section, section 8.

Related work
Word Sense Disambiguation is one of the hardest problems in NLP. Successful supervised WSD approaches (Lee et al., 2004;Ng and Lee, 1996) are restricted to resource rich languages and domains. They are directly dependent on availability of good amount of sense tagged data. Creating such a costly resource for all language-domain pairs is impracticable looking at the amount of time and money required. Hence, unsupervised WSD approaches (Diab and Resnik, 2002;Kaji and Morimoto, 2002;Mihalcea et al., 2004;Jean, 2004;Khapra et al., 2011) attract most of the researchers. Khapra et al. (2011) dealt with bilingual unsupervised WSD. It uses EM algorithm for estimating sense distributions in comparable corpora. Every polysemous word is disambiguated using the raw counts of its translations in different senses. Synset aligned multilingual dictionary (Mohanty et al., 2008) is used for finding its translations. In this dictionary, synsets are linked, and after that the words inside the synsets are also linked. For example, for the concept of 'boy', the Hindi synset {ladakaa, balak, bachhaa} is linked with the Marathi synset {mulagaa, poragaa, por}. The Marathi word 'mulagaa' is linked to the Hindi word 'ladakaa' which is its exact lexical substitution.

Background
Suppose words u in language L 1 and v in language L 2 are translations of each other and their senses are required. The EM based formulation is as follows:
• crosslinks L 1 (a, S L 2 ) is the set of possible translations of the word 'a' from language L 1 to L 2 in the sense S L 2 .
• π L 2 (S L 1 ) means the linked synset of the sense S L1 in L 2 .
E and M steps are symmetric except for the change in language. In both the steps, we estimate sense distribution in one language using raw counts of translations in another language. But this approach has following limitations: Poor performance on verbs: This approach gives poor performance on verbs (25%-38%). See section 6. Same sense throughout the corpus: Every occurrence of a word is tagged with the single sense found by the algorithm, throughout the corpus. Closed loop of translations: This formulation does not work for some common words which have the same translations in all senses. For example, the verb 'karna' in Hindi has two different senses in the corpus viz., 'to do' (S 1 ) and 'to make' (S 2 ). In both these senses, it gets translated as 'karne' in Marathi. The word 'karne' also back translates to 'karna' in Hindi through both its senses. In this case, the formulation works out as follows: The probabilities are initialized uniformly. Hence, P (S 1 |karna) = P (S2|karna) = 0.5. Now, in first iteration the sense of 'karne' will be estimated as follows (E-step): Similarly, in M-step, we will get P (S 1 |karna) = P (S 2 |karna) = 0.5. Eventually, it will end up with initial probabilities and no strong decision can be made.
To address these problems we have introduced contextual clues in their formulation by using semantic relatedness.

Modified Bilingual EM approach
We introduce context in the EM formulation stated above and treat the context as a bag of words. We assume that each word in the context influences the sense of the target word independently. Hence, where, w is the target word, S is one of the candidate synsets of w, C is the set of words in context (sentence in our case) and c i is one of the context words.
Suppose we would have sense tagged data, p(S|w, c) could have been computed as: But since the sense tagged corpus is not available, we cannot find #(S, w, c) from the corpus directly. However, we can estimate it using the comparable corpus in other language. Here, we assume that given a word and its context word in language L 1 , the sense distribution in L 1 will be same as that in L 2 given the translation of a word and the translation of its context word in L 2 . But these translations can be ambiguous, hence we can use Expectation Maximization approach similar to (Khapra et al., 2011) as follows:

E-
Step: x ∈ crosslinks L 2 (u, S L 1 i ) crosslinks L 1 (a) is the set of all possible translations of the word 'a' from L 1 to L 2 in all its senses. σ(v, b) is the semantic relatedness between the senses of v and senses of b. Since, v and b go over all possible translations of u and a respectively. σ(v, b) has the effect of indirectly capturing the semantic similarity between the senses of u and a. A symetric formulation in the M-step below takes the computation back from language L 2 to language L 1 . The semantic relatedness comes as an additional weighing factor, capturing context, in the probablistic score. M-Step: a) is the semantic relatedness between the senses of u and senses of a and contributes to the score like σ(v, b).
Note how the computation moves back and forth between L 1 and L 2 considering translations of both target words and their context words.
In the above formulation, we could have considered the term #(word, context word) (i.e., the co-occurrence count of the translations of the word and the context word) instead of σ(word, context word). But it is very unlikely that every translation of a word will co-occur with

Computation of semantic relatedness
The semantic relatedness is computed by taking the inverse of the length of the shortest path among two senses in the wordnet graph (Pedersen et al., 2005). All the semantic relations (including crosspart-of-speech links) viz., hypernymy, hyponymy, meronymy, entailment, attribute etc., are used for computing the semantic relatedness. Sense scores thus obtained are used to disambiguate all words in the corpus. We consider all the content words from the context for disambiguation of a word. The winner sense is the one with the highest probability.

Experimental setup
We have used freely available in-domain comparable corpora 1 in Hindi and Marathi languages. These corpora are available for health and tourism domains. The dataset is same as that used in (Khapra et al., 2011) in order to compare the performance.

Error analysis and phenomena study
Our approach tags all the instances of a word depending on its context as apposed to basic EM approach. For example, consider the following sentence from the tourism domain: х । (vaha patte khel rahe the) (They were playing cards/leaves) Here, the word (plural form of ) has two senses viz., 'leaf' and 'playing card'. In tourism domain, the 'leaf' sense is more dominant. Hence, basic EM will tag with 'leaf' sense. But it's true sense is 'playing card'. The true sense is captured only if context is considered. Here, the word х (to play) (root form of х ) endorses the 'playing card' sense of the word . This phenomenon is captured by our approach through semantic relatedness.
But there are certain cases where our algorithm fails. For example, consider the following sentence: (vaha ped ke niche patte khel rahe the) (They were playing cards/leaves below the tree) Here, two strong context words Ú (tree) and х (play) are influencing the sense of the word . Semantic relatedness between Ú (tree) and (leaf) is more than that of х (play) and (playing card). Hence, the 'leaf sense' is assigned to . This problem occurred because we considered the context as a bag of words. This problem can be solved by considering the semantic structure of the sentence. In this example, the word (leaf/playing card) is the subject of the verb х (to play) while Ú (tree) is not even in the same clause with (leaf/playing cards). Thus we could consider х (to play) as the stronger clue for its disambiguation.

Conclusion and Future Work
We have presented a context aware EM formulation building on the framework of Khapra et al (2011). Our formulation solves the problems of "inhibited progress due to lack of translation diversity" and "uniform sense assignment, irrespective of context" that the previous EM based formulation of Khapra et al. suffers from. More importantly our accuracy on verbs is much higher and more than the state of the art, to the best of our knowledge. Improving the performance on other parts of speech is the primary future work. Future directions also point to usage of semantic role clues, investigation of familialy apart pair of languages and effect of variation of measures of semantic relatedness.