{'fraction_non_alphanumeric': 0.03715249061645143, 'fraction_numerical': 0.010751320058527896, 'mean_word_length': 4.8093126385809315, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fast developing of a Natural Language Interface for a Portuguese WordNet: Leveraging on Sentence Embeddings

We describe how a natural language interface can be developed for a wordnet with a small set of handcrafted templates, leveraging on sentence embeddings. The proposed approach does not use rules for parsing natural language queries but experiments showed that the embeddings model is tolerant enough for correctly predicting relation types that do not match known patterns exactly. It was tested with OpenWordNet-PT, for which this method may provide an alternative interface, with benefits also on the curation process.


Introduction
A natural way of interacting with computational systems or knowledge bases is to use the same language we use for interacting with other humans. However, due to all the complex phenomena of natural language, most systems rely on browsing, keyword-based search interfaces or their combination. This is simpler at the technical level and avoids having to deal with Natural Language Understanding issues. The previous phenomena include ambiguity and language variability and are the reason why matching natural language with formal queries is not a trivial task. To overcome this challenge, we investigate how a model of sentence similarity can be exploited by a natural language interface (NLI) for a wordnet. Our approach is tested in OpenWordNet-PT (de Paiva et al., 2012) (OWN-PT), probably the most active Portuguese wordnet (de Paiva et al., 2016b).
The development of this system, dubbed NELIO, requires only a small set of handcrafted templates for each query to be covered. Instantiating those templates with arguments from OWN-PT results in a large set of sentences, used for training a doc2vec (Le and Mikolov, 2014) model. The latter is a variation of the popular word2vec (Mikolov et al., 2013) but, besides learning dense vector representations of words, it learns a representation for documents (sentences, in our case), based on the words used and on a document label. Such a model can be used e.g., for predicting the most suitable label for an unseen document. In this work, we rely on the trained doc2vec model for predicting the relation type that a natural language query is asking for. We then use this information for querying OWN-PT and retrieving suitable answers. This process is fast enough and avoids writing a set of rules for parsing natural language queries. Besides providing a more natural way of interacting with OWN-PT, NELIO turns out to be an alternative way of exploring OWN-PT and reveal flaws that, otherwise, would not be easy to spot.
The remainder of this paper briefly overviews OWN-PT, describes the development of NELIO, reports on performed experiments, including a systematic evaluation of the model in this context, and, before concluding, overviews related work.

OpenWordNet-PT
OpenWordNet-PT (OWN-PT) is an ongoing project to build a wordnet for Portuguese. It is aligned with Princeton WordNet (Fellbaum, 1998) (PWN), but still has about half of its size. So far, only partial evaluations of its coverage were performed, namely of verbs (de Paiva et al., 2016a, nouns , and (gentilic) adjectives .
OWN-PT is freely available in RDF/OWL. Its data can be retrieved via a SPARQL endpoint, but it can also be explored through its own web interface 1 or through the interface of the Open Multilingual WordNet (Bond and Foster, 2013). As previously suggested (Real et al., 2015), a visual interface helps to discover interesting issues to work on. The research presented here is related to lessons previously learned.
3 System Development NELIO interprets questions, in Portuguese, that ask for concepts, lexicalised as y, which are related in some way to another concept lexicalised as x, mentioned in the question. This section describes the steps for developing its current version.

Question Templates
To enable the generation of prototypical questions, a small set of templates for each covered relation was handcrafted by the first author of this paper. Such templates generalise possible ways of asking the desired questions in Portuguese. All templates currently used (between 3 and 10 per relation) are revealed in table 1, grouped according to the target relation. Most semantic relations in OWN-PT are covered. Yet, due to their different scope, lexical relations were left out of this set.

Model Training
The generation of prototypical questions results from filling the templates, automatically, with real examples from OWN-PT. Those questions were used to train a doc2vec (Le and Mikolov, 2014)  The learned model can be exploited in a classification task. More precisely, given a fragment of text, it can be used for predicting the appropriate label. Once predicted, the label is used together with the relation argument that appears on the question (x) for generating a SPARQL query, which can be made to OWN-PT for retrieving the possible answers.

Fixed Argument Extraction
Besides classifying the question into a relation type, the fixed relation argument x must be extracted from the input text. In all handcrafted templates, this argument is the last term of the question. In fact, for the type of considered questions, there would not be many variations where this was not the case. Therefore, the extraction of x was simplified in such a way that it is always the last sequence of words in the question. More precisely, in order to cover multiword expressions, the system searches for the longest lexical form in OWN-PT starting with the i th , i ∈ (1, n], and ending in the last token of the question. For instance, given the question que tipos há de intoxicação alimentar? (what types are there of food poisoning?), the system checks, in the following order, whether OWN-PT covers the forms: tipos há de intoxicação alimentar, há de intoxicação alimentar, de intoxicação alimentar, intoxicação alimentar. It stops once it finds that the lexical form intoxicação alimentar (food poisoning) exists.

SPARQL Generation
With the label and the fixed argument, a SPARQL query can be generated to get all the valid lexical forms for y. Figure 1 shows the generated query for the question que formas há de correr?, with label [hyponymOf] and x = correr. It retrieves lexical forms (lf ) in OWN-PT synsets (s2) for which the aligned PWN synset (sen2) is a hyponym of another PWN synset (sen1) that is aligned with an OWN-PT synset with the lexical form correr.

Examples
The resulting doc2vec model identifies the correct relation type in most situations. Besides being easy to train, a good thing about it is that no syntactic analysis is required and, still, the text of the questions does not have to match the original templates exactly. This provides an interesting level of tolerance while dealing with syntactic variability. Table 2 shows some of the questions answered correctly that, despite sharing some words, do not perfectly match any template.
On the other hand, since the model is based on the surface forms of words, it does not handle variation of vocabulary. For instance, the question "qualé a matéria de chocolate?" (what is the matter of chocolate?) is answered with hyponyms of 'chocolate', instead of its substances, because no substanceHolonymOf template includes the word matéria (matter). However, this situation can be easily fixed by adding a single template with this word for the previous relation, e.g., matéria-prima de <Y>?.

Generalisation
Combining all possible question templates with all the relation instances in OWN-PT results in 298,249 different questions. Those questions were generated, shuffled and used as a dataset for training and testing NELIO.
In a simple experiment for analysing how well  the model generalises, it was tested with different proportions of training and testing data. Table 3 presents the accuracy, i.e., the proportion of questions correctly answered in this experiment. This also showed that, some of the incorrect answers were in fact empty, due to misclassification of the relation type, which suggested a second experiment: similar to the previous but, when the given answer was empty, NELIO tried to get an answer with the second or third relation type predicted by doc2vec. As expected, this resulted in higher accuracies, also in table 3 (Top-3).  When considering only the top label, training the model with 90% (≈268k), 50% (≈149k), or even 20% (≈59k) of the questions, results in accuracies above 90%. This happens mainly because, although there are only a few templates, they are instantiated many times. With lower training proportions, accuracy drops more considerably. Yet, with only 5% it is still close to 80%.
Accuracy is different for different relations. For instance, with 90% of training data, it ranges from 100%, for entails, antonymOf and hyponymOf, to 73%, for substanceMeronymOf. A closer look shows that, except for the meronym-holonym relations, all accuracies are higher than 94% (hyper-nymOf). The problem with the former is that they are very similar and, for this reason, share several templates among them, which confuses the model.
The aforementioned issue is significantly minimised when the top-3 labels are considered. In this case, accuracies are 97% or higher with 15% or more training data. Specifically, they are 98% or higher for all relation types, except for the meronym-holonym, which are still the most problematic. The lower accuracy in this scenario is for memberHolonymOf (87.9%).

Related Work
Traditional Automatic Question Answering (QA) follows an Information Retrieval perspective (Kolomiyets and Moens, 2011). Queries are typically natural language questions (NLQs) and answers are retrieved from a collection of written documents. But the development of natural language interfaces (NLIs) for databases has also been a research topic for a long time (Androutsopoulos et al., 1995). Here, the primary challenge involves translating NLQs to formal queries made to a database. Knowledge-based QA systems are a specific case of the previous.
Several NLIs for ontologies -e.g., Querix (Kaufmann et al., 2006), PANTO (Wang et al., 2007), FREyA (Damljanovic et al., 2010) -translate NLQs to SPARQL with a set of rules on the result of syntactically parsing NLQs, possibly using PWN for synonym expansion. A similar approach (Unger et al., 2012) may be based on SPARQL templates, to be filled with entities and predicates identified in the NLQ.
Other systems rely on domain-independent semantic parsers that learn how to map NLQs to predicates in a large knowledge base, based on question-answer pairs. SEMPRE (Berant et al., 2013) maps words to predicates and then combines the predicates to the final logical form. Another possibility (Kwiatkowski et al., 2013) is to parse utterances for producing an underspecified logical form, before mapping lexical predicates to the target ontology predicates. The previous systems were assessed while resorting to Freebase for answering NLQs. Yet, as opposing to Freebase or DBPedia, wordnets have a much smaller number of predicates. So, it could be worth exploring how semantic parsers could be adapted for our work.
Once translated to SPARQL, generally to a subset of this language, expressiveness is limited. To avoid this, SQUALL (Ferré, 2014) is a controlled natural language for querying and updating RDF datasets. Nouns and intransitive verbs are used as classes; relation nouns and transitive verbs as properties; and proper nouns as resources. Syntactic and semantic analysis is implemented as a Montague grammar, an approach that would work for querying a wordnet, considering the simplicity of its RDF model. On the other hand, SQUALL requires that end-users comply with its controlled syntax, and know the RDF vocabulary.
An alternative approach (Bordes et al., 2014) learns low-dimensional embeddings of words and entities, respectively in questions and relation types of Freebase. This way, representations of questions and of their corresponding answers are close to each other in the joint embedding space. More recent works (Neelakantan et al., 2016;Zhong et al., 2017) rely on neural networks for translating NLQs to formal queries, thus avoiding domain-specific grammars or rules.

Conclusion
We have described how we can leverage on sentence embeddings in the development of a NLI for a wordnet. The proposed procedure was applied to OWN-PT with some success. When trained in a subset with at least 20% of the possible questions, generated with a small set of templates, and tested with the remaining questions, accuracies were higher than 91%, when using the first pre-diction, or 97%, when trying with the first three predictions, in case the previous did not return an answer. This simple experiment confirmed that the proposed approach works well with the doc2vec model for predicting the correct relation type. Despite the positive results, this experiment revealed that the system is confused by similar relations, for which the templates share vocabulary, namely the three types of meronymy. The problem can be minimised by considering the top-3 predictions, but others, such as merging the three relations, can be analysed in the future.
Still, this was a limited experiment, where known limitations of the system had a low impact. This includes questions with vocabulary not covered by the templates, or questions that do not end with the fixed word. The former can be minimised by adding alternative templates. The second is due to a simplification that works for many cases, but fails for some, as in the question 'quais frutas existem?' (what fruits exist?), where the target word is frutas. The previous question has to be made like 'quais os tipos de fruta?'. In the future, we will devise more general ways of extracting the target argument from the question, e.g., having in mind that, among the words/expressions in the question, it should be the least frequent in the dataset; or maybe training an automatic sequence labeller for identifying the target argument in the context of a question. In the latter case, training data should also include templates that do not end with the target argument.
Other possible directions for future work include: (i) Presenting the answers according to the senses they apply to, because context is not enough for disambiguation (currently, there is an option for considering only the first sense); (ii) Adding alternative types of question e.g., what is the relation between x and y ? or is y related to x ?, to be answered, respectively, with the name of a relation between x and y in OWN-PT, or yes/no, depending on the existence of such a relation; (iii) Exploring recent models for representing sentence meaning, learned from natural language inference data (Conneau et al., 2017), though available data in Portuguese (Fonseca et al., 2016;Real et al., 2018) may not be enough.
Despite its limitations, NELIO was already helpful for finding issues in OWN-PT that need to be fixed. It showed flaws such as inconsistencies in the capitalization (e.g., Salmão, Pardal), presence of underscores instead of spaces (e.g., animal doméstico), or plural instead of singular form (e.g., epidemias, montanhas), not to mention actual errors (e.g, dançar entails andar, in English, dancing entails walking).
A mid-term goal is to make NELIO available from a web interface. In the meantime, its source code is available online, at https://github. com/hgoliv/nli_openwordnet-pt. Although, so far, the proposed approach was only used as a NLI for a wordnet, in principle, a similar approach could be used in the development of a NLI for any knowledge base represented as a-relatedTo-b triples.