{'fraction_non_alphanumeric': 0.03455453788509576, 'fraction_numerical': 0.007771301693033583, 'mean_word_length': 4.746810207336523, 'pattern_counts': {'":': 14, '<': 2, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ambiguss, a game for building a Sense Annotated Corpus for French

This paper presents Ambiguss, a Game With A Purpose designed both to collect ambiguous sentences and build a Sense Annotated Corpus. It also generates a lexicon of polysemous words associated with the glosses that illustrate the different meanings. Early evaluations indicate that the approach is relevant and efficient.


Introduction
Evaluating a WSD task is a challenge at least as difficult as developing the task itself. Manually constructing a corpus of ambiguous sentences is a difficult and tedious task. Such a corpus is even more complex to produce if in each sentence, each ambiguous word is associated with its correct meaning. In addition to finding/imagining the sentence, it is necessary to have a word sense lexicon to associate the correct meaning with each ambiguous word, and such a resource may not be available in any given language. Secondly, referring to a given meaning might be tricky. This can be done by associating a meaning number, but in such a case, the lexicon must be provided along with the corpus. Another method would be to represent the correct meaning of the ambiguous word by a gloss, i.e. a word or group of words that intuitively refers to its correct meaning. A classic example could be bank > river and bank > money; the glosses river and money refer to two possible meanings of the word bank. Anyway, to define the glosses, i.e. to choose the word that best illustrates the meaning of a word is another pitfall: Indeed, it seems that there is not always a strong agreement between people for such a task.
Collect lexical data by crowdsourcing through games is a new trend since few years. The question that arises is to properly identify what kind of data can be collected in the domain of Natural Language Processing and Semantic computing. The postulate is that the semantic information understandable by the native speakers can be collected in this way. Furthermore, the type of semantically ambiguous sentence in which we are interested here, refers mostly to common sense knowledge (to be tackled with knowledge bases in the spirit of (Lenat, 1995)). This type of knowledge is intuitive and easy to infer for speakers, but definitively difficult to model to develop an Artificial Intelligence that would perform properly WSD. This paper presents the design of a Gwap (Game With A Purpose) called Ambiguss, whose goal is twofold. On one hand, this game allows collecting ambiguous sentences created by players. On the other hand, these sentences are proposed to other players who must select from a list of glosses the one that corresponds to the correct meaning of each ambiguous word of the sentence. First, we show that the concept of Gwap is adapted to the production of Sense Annotated Corpus. Then we describe a novel Gwap designed to build a Sense Annotated Corpus. We conclude with a shallow evaluation of the first data collected.

From GWAPs to Sense Annotated Corpora
How could we relate and apply the idea of Gwap to the production of a Sense Annotated Corpus?
With the advent of Gwaps and more generally crowdsourcing approaches, one observe a fundamental shift in language resources building that is going on now (see (Chamberlain et al., 2013) for more details). If the use of crowdsourcing microworking platforms like Amazon Mechanical Turk can be questionable, both in terms of ethics and quality , Gwaps represent a paradigm shift for the acquisition of lexical resources, which passes from hand-made by specialists to crowdsourced through entertainment by native speakers. However, creating an efficient GWAP involves principles and developments that are not that straightforward.
The concept of Gwap was first proposed by Dabbish, 2004 and2006) and is based on the idea that beyond the simple entertainment of players, a game can have a purpose, such as data acquisition or problem solving. Since then, the first principles defined by (von Ahn et al., 2008) have given rise to Gwaps in various fields (see Lafourcade, et al., 2015 for a survey). In (Chamberlain et al., 2009), the Phrase Detectives game is proposed to solve coreferences in texts, which is conceptually quite close to construct a corpus of sense annotated sentences. Wordrobe (Venhuizen et al., 2013) is a game for labeling word senses, but senses come from a predefined set (the Groningen Meaning Bank).
Besides being funny and addictive, the mandatory characteristics for a Gwap to be successful could be found by asking simple questions: what can we ask to players? How could we ask these things to players? The task of selecting the proper meanings of words contained in sentences is clearly within the reach of native speakers. But one may wonder if this is a fun enough task to give rise to a game, and entertaining enough so that players are willing to do it?
According to (Habert et al., 1998) a corpus is a collection of language related data which are selected and organized according to linguistic criteria so as to be used as samples (for evaluation, for instance). Amongst those corpora, there are the British National Corpus (Burnard, 1998) (100 million words) and the American National Corpus (Ide & Macleod, 2001) (20 million words). In a Sense Annotated Corpus each word is related to a word meaning, usually by means of a sense number linked to a specific database (see Vial et al., 2017). For example, the corpus of the task number 7 of SemEval 2007 is annotated with the identifiers of Princeton WordNet 2.1 (Fellbaum and Miller, 1998). The English corpus for the task 13 of SemEval 2015 2015 (Moro & Navigli, 2015) is annotated with the identifier of version 3.0 of BabelNet. Building such corpus might be based on various motivations, the most obvious of them being to provide a basis for automatically evaluating WSD systems.
For the French language, there is no such corpus freely available. Furthermore, there is a strong need for a corpus that would not be directly dependent on a given database. Hence the interest of indicating the meaning of words by glosses rather than identifiers related to specific meanings of a specific resource. Evaluating the feasibility of developing a new GWAP designed to construct such a corpus sounds interesting. The main issues lie in the design of the game which must be addictive and efficient, and secondarily in the evaluation of the collected data.

Ambiguss, a game to collect Semantically Ambiguous Sentences
Ambiguss (https://ambiguss.calyxe.fr/) is a game to solve lexical ambiguities in sentences, but also to collect such semantically ambiguous sentences (SemAS). Although this might be the case with some collected sentences, Ambiguss is not designed to collect syntactically ambiguous sentences. Everything is designed so that players do only need common language knowledge. The game is aimed at native speakers, who are not expert in linguistics or grammar.

Playing ambiguous sentences
Once logged, the player is proposed a random sentence in which at least one word is highlighted. The purpose of the game is to select the proper meaning of each highlighted word, through an user interface menu that presents a list of several possible meanings. When done, the player has to click on the Valider (Eng. Validate) button, and then he gets the result (Figure 2). There is no time constraint in the game.
In Figure 1, an example of game is given. The words rat and radis are highlighted: this means they are polysemous and the player is invited to disambiguate them by choosing the correct meaning according to the sentence, as seen in the figure for the word rat, which has two possible meanings.  Players earn two types of rewards, points, only for fame, and credits that can be used for typical game actions, such as creating oneself an ambiguous phrase or adding a gloss: indeed, during the game, a player may add his own gloss, if he feels that none of the proposed ones is appropriate. The amount of points earned is related to the distribution of other player answers. (Of course, the main hypothesis, which is verified experimentally, is that globally, players provide the adequate answers).

Creating ambiguous sentences
The opportunity to create an ambiguous sentence is both a reward for good players and an incentive to play well, while being a way of spending the credits earned by playing. To do this, the player has to select the "Créer" menu and key in a sentence. He/she can then select one or more word(s) in this sentence and declare them as ambiguous. If an ambiguous word does not yet have glosses, he/she adds some to it to establish a list of possible meanings, and indicates what the correct gloss is for the term in the context of his/her sentence. These ambiguous words are those that will be highlighted and that another player will have to disambiguate in a game.
Although the user interface was deemed intuitive by the players, for now, only a small fraction of them (15%) has created sentences. It would seem that the fear of being judged through the quality of his own sentences might be deterrent to some players. Choose the correct gloss for each ambiguous word.

Why creating sentences? For greed and fame of course!
Each time a player earns points and credits, the creator of the sentence recovers 10% of the total. Hence, from a strict gaming point of view, players have a definite interest in creating sentences with many interesting ambiguities. When he/she plays, a player can pass over a sentence, if he/she finds it boring or nonsensical. Moreover, when playing, people can like a sentence (and share it on Facebook and Twitter). Having a high number of likes is an incentive for many players. Hence, they trend to produce interesting and funny sentences. Some of those sentences are "undecidable" in the absence of context, for example:

Je suis une fille. L'avocat est véreux La petite brise la glace Je loue un appartement
In such cases, the interest of the game is to make emerge the preferred meaning, the one chosen by the majority of the players. But in some cases, a quite even number of votes for glosses is also possible.
The game proposes a multi-criteria ranking that orders players according to various parameters (number of points, number of sentences created, and number of likes obtained). Another ranking concerns the sentences and orders them according to the number of likes or the gains they have given to their creator. All this is designed to flatter and retain the player and thus intensify his contribution.

About the data collected
Ambiguss is still a prototype, but people have been asked from the beginning to serve as testers. By word of mouth, we reach after few weeks a total number of 64 players. Most of them have just played the disambiguation game, few daring to embark on the creation of sentences. A manual evaluation by expert in NLP concluded that 96% of the sentences are valuable for WSD evaluation as they are mostly good examples of semantic ambiguity related to common sense. Are the sentence properly disambiguated? Are the collected glosses correct? So far, 100% of the sentences are disambiguated properly. More precisely, the gloss predominantly selected is the one that corresponds to the correct meaning in 100% of the cases. If we go into details, players achieve collectively a precision of 0.985. This means that it is very rare for a player to select a wrong meaning. Recall is not applicable in this context. Wrong glosses for a given term are never selected, but some glosses (4%) are not very representative of the meaning to be illustrated.

Reporting incorrect data
Players have access to an user interface button to indicate an incorrect item, whether it's a spelling mistake in a sentence or a gloss, a bad gloss, an absurd phrase, an ethical violation, and so on. The reporting mechanism allows self healing of the data and is very fast in practice (wrong data do not stay very long uncorrected).

Exporting the data
The data collected with Ambiguss are freely available to anyone. Two sets of data are available: the sentences, and the list of ambiguous terms with their glosses. The export format is JASON. For example, exporting the sentence C'était un vol agréable malgré la distance (Eng. It was a pleasant flight/theft despite the distance.) produces the following output : {"phrase":"C'était un <amb id=\"1\">vol<\/amb> agréable malgré la distance.","reponse":[{"motAmbigu":"vol","ordre":1,"nbRep":30,"gloses":[{"valeur":"crime","nb Rep":"1"},{"valeur":"voyage","nbRep":"29"}]}]} The sentence is given with inside annotations for indicating the ambiguous words. The field "réponses" (Eng. Answers) gives for each ambiguous word the distribution of the glosses proposed by the players. In the above example, 29 players answered "voyage" for the word "vol" and only 1 player proposed "crime". Of course, this particular sentence can hold at least two interpretations, but only one sounds reasonably meaningful. Concerning the lexicon of terms and glosses, the export is also under a JASON form: {"motAmbigu":"vol","gloses":["voyage","crime"]} {"motAmbigu":"souris","gloses":["animal","objet","gigot","périphérique informatique","nana"]} For the word souris, 5 glosses are proposed. It is most than probable that in many cases more glosses are proposed than the actual number of meanings. That is to say, for some meaning, several glosses are present. So, there would remain a task not foreseen by the game: to distribute the glosses for each polysemous word in equivalence classes. For example, in the case of souris, both glosses « périphérique informatique » and « objet » might probably refer to the same meaning.

Conclusion
We presented a game, called Ambiguss, designed for collecting ambiguous sentences along with player choices for the proper meanings. Another output of the game is that it builds a lexicon of polysemous words with their associated glosses. Although still in its testing phase, the game was judged highly addictive by many testers.
We have already collected several hundred semantically ambiguous sentences (for French). This corpus can be freely used as a reference for easily evaluating an automatic WSD process (or any relevant tasks). Word meanings are represented as glosses, which avoids the use of a dedicated semantic lexicon. The list of polysemous words and glosses is also freely accessible.