{'fraction_non_alphanumeric': 0.02839373793695046, 'fraction_numerical': 0.01110872828651083, 'mean_word_length': 5.1715193223928, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Study of Similarity Measures as Features in Classification for Answer Sentence Selection Task in Hindi Question Answering: Language-Specific v/s Other Measures

Answer sentence selection is an important sub-task in Question Answering (QA) that determines the correct answer sentence from a passage. This task can naturally be reduced to the semantic text similarity problem between question and answer candidate. In this work, we investigate the significance of various similarity measures for the answer sentence selection task in Hindi an Indo-Aryan language. Karaka relations is the core of dependency annotation scheme used for Hindi and are crucial to syntactico-semantic analysis of the sentence. We investigate this, and compare them to other, hitherto known measures. To investigate and compare the utility of various measures, we develop a test-bench over a benchmark Hindi and English multilingual QA corpus for comparison, making two tool-chains and designing empirical experiments across combinations of similarity measures, sentence embedding schemes, and supervised machine learning models for classification. Combining Karaka relations with different similarity measures shows significant performance improvement for sentence selection task, suggesting them as potentially a semantic similarity measure. Moreover, our results give us confidence that refinement of Karaka relations extraction to optimal quality will reduce the need for availability of large pre-trained language models.


Introduction
Information-retrieval-based (IR-based) QA systems typically employ a search engine and a passage retrieval module that narrows down the answer search space from a huge corpus to a small set of sentences. On the contrary, QA systems depending on reading comprehension are provided with a passage for answer extraction. In both kinds of systems, the returned answer can be a word, a phrase or a sentence. Wang et al. (2007) highlighted the importance of returning a complete sentence over a short phrase as an answer against the user query. The study conducted by  shows that as the amount of text returned by a QA system increases, users utilize the supporting text to get the answer, significantly decreasing the number of further queries.
Answer sentence selection as an important task in QA has been a widely researched topic since the release of QASent and WikiQA corpora. QASent (Wang et al., 2007) was created by choosing sentences from Text REtrieval Conference (TREC) QA 8-13 track data (Voorhees, 1999). WikiQA (Yang et al., 2015), that was released later, was developed in a more natural and large setting. Researchers have sought to accomplish this task by combining (in varying degrees) techniques of natural language processing, statistical analysis and deep learning for lexical, syntactic and semantic processing of the question and candidate sentences. We can certainly say that the performance of QA systems depends largely on the quantitative and qualitative advances in computational resources specifically available for the language of QA. But then this is a key challenge for low resource languages. Languages that have less digital presence, languages in which colloquiality and dialects dominate standardized printed and digital content, languages with complicated descriptive grammars elusive to the usual parsing schemes are such low resource languages. Many Indo-Aryan languages (mainly spoken and used in South Asia or SAARC countries) lack sizeable, curated datasets and efficient, effective computational linguistics tools. The resources available for the language-dependent processing necessary for the answer sentence selection task described above are paltry and fragmented, mostly dependent on datasets translated from English. We observe that there is need of more efficient, less-computation-intensive, less-data-dependent computational linguistics tools for these languages.
Indo-Aryan languages have relatively free word order compared to European-origin languages (Sangal and Chaitanya, 1995). They have a rich system of case-endings and post-positions (together called vibhakti). For the important task of answer selection in these languages, we need to choose measures that take into account their distinguishing aspects. Relatively lesser attempts have been made for QA task in Hindi as well as in other Indo-Aryan languages.
In this work, the answer sentence selection task for QA in Hindi is addressed. We explore various lexical and syntactico-semantic similarity measures between question and candidate answer sentence for accomplishing this task. We regard this as a classification problem in our work and investigate the influence of similarity measures on the answer sentence selection task in Hindi. For comparison, we perform empirical analysis over Hindi and English multilingual QA corpus, experimenting across combinations of seven similarity measures, two sentence embedding techniques and eight machine learning algorithms.
The paper is organized as follows: In the next section 2, we survey the related work for English along with development for QA in Indian languages. Section 3 defines the problem statement along with description of various similarity measures explored and evaluated. Experiment design and results on benchmark Hindi and English multi-lingual QA datasets are presented in Section 4. Conclusions and future work are discussed in the last section 5.

QA in Indian languages
Since 2000, Hindi-English cross-lingual question answering systems have been demonstrated. As a part of the trans-lingual information detection, extraction and summarization program Satoshi et. al (Sekine and Grishman, 2003) developed a system that accepts an English query, searches the answer in Hindi newspapers and returns an English response. (Shukla et al., 2004) created an intermediate representation called Universal Networking Language(UNL), that makes a hyper-graph to represent the meaning of the text in source language. Thus given a source text in several languages, their system could respond without translating the source text in questioner's language. (Gupta et al., 2012) proposed a natural language interface to relational database using computational Paninian grammar and Karaka relations to perform semantic analysis. (Gupta et al., 2018) proposed a framework for cross-lingual factoid and descriptive QA in Hindi and English. They deployed a deep learning model for question classification and answer extraction using similarity measures like proximity score, pattern matching score, n-gram coverage score and semantic similarity score.

Answer Sentence Selection
Studies highlight that measures like overlapping ngrams, surface pattern matching or bag-of-word representations give certain level of success in answer sentence selection (Martinez et al., 2012), while they fail to determine syntactic and semantic variations between question and answer pair which should be captured for appropriate answering (Yih et al., 2013). In many QA systems syntactic relations between verb, arguments and certain type of question words are extracted from the parse tree (Shen et al., 2005). Following the approach pioneered by Punyakanok et al. (2004), question and candidate sentences were represented with their dependency tree that incorporate semantic information along with using tree-edit distance between them as selection criteria (Wang and Manning, 2010;Heilman and Smith, 2010;Shen and Klakow, 2006). Ignoring important relations between tokens in a sentence is a reason for false positives for the existing retrieval techniques (Tellex et al., 2003) This was mainly because several irrelevant passages may share the same query keywords, but the relations between these tokens might be totally varying from relations in query keywords. (Cui et al., 2005) demonstrated that to determine the similarity between two sentences it is crucial to examine similarity between all the corresponding relation paths extracted from dependency trees. (Sultan et al., 2014)'s work highlights that dependency types may exhibit equivalence and it is necessary to develop a mapping between dependency types. Calculating best match between syntactic and semantic structured representations incurs high computational cost O(V 2 L 4 ) (Yih et al., 2013). Later works explored named entity recognition, answer type tagging, word and phrase alignments (Sutedi et al., 2019), (Wang et al., 2007). It has been observed that performance and errors from each of the above mentioned syntax and semantic analysis modules impacts the answer selection accuracy performance. More recently, deep learning approaches tend to have overshadowed these NLP tasks that do not rely on linguistic tools, but certainly require huge corpus and computational resources (Tan et al., 2016) (Ma et al., 2015) described dependency-based convolution neural networks (CNN) which gave higher performance for question classification tasks over the baselines. Current efforts are towards learning sentence representation that captures essential information to strengthen similarity between question and candidate sentence pair (Feng et al., 2015), (Yu et al., 2014). Large scale pre-trained word vectors and multi lingual models for sentence representations have been released for some group of languages (Artetxe and Schwenk, 2019), (Sanh et al., 2020). Several baselines have been established for answer sentence selection task in English over bench mark dataset like TREC-QA, WikiQA, QASent. To the best of our knowledge this is a first attempt for the key task of answer sentence selection in Hindi QA.

Problem Definition
Given question q and a passage/context with a a set of candidate answer sentences {s 1 ,s 2 ... s n } for q, the problem is to identify the best matching candi-date sentence s i where (1 ≤ i ≤ n). If the sentence identified is the ground truth answer then q is considered as correctly answered. We formulate this as a classification problem in a supervised learning setting. Each instance in the training corpus is a context and question pair, associated with a target label which indicates the index of the answer sentence within the context.Two such instances are shown in figure 1. Using this labeled dataset we learn an answer sentence selection model that predicts the target label for any new instance of context and question pair. To train this machine learning model, we transform the given candidate sentences from context to features, value of which is based on its similarity to question. The focus of this work is to evaluate the similarity variants based on model's accuracy to predict correct answer sentence from the given context. The high-level process for experimenting with different similarity variants is depicted in figure 2.

Similarity Measures
Given an instance of question and a context with n candidate answer sentences (candidates), we represent it using n-dimensional feature set, in the training set, the value of which is computed using similarity between the question and candidate. For each candidate, we build one feature based on its similarity to question. If a context has less than n sentences, we replace its feature value with a score that indicates no correlation between question and candidate, to make total n candidates in all contexts for uniformity. In order to compute similarity between question and candidate, we use seven different lexical and syntactico-semantic measures either individually or in combinations of each other.

Content Word Overlap (cwo)
The simplest measure is lexical overlap between the words of the question and candidate sentences (Gupta et al., 2018;Yih et al., 2013). For certain datasets like QASent (Wang et al., 2007), where candidate answer sentences for questions are curated by matching content words of question. Yang et al. (2015) show that simple word matching method establishes a strong baseline. However the performance may degrade for datasets arising in more natural and realistic situations. We explore this pos-

Longest Common Sub-string (lcs)
Longest Common Sub-string is the maximum length sub-string from all common sub-strings between two texts. In this work, we calculate the length of longest common sub-string between question and the candidate answer sentences to compute similarity.

Cosine Similarity (cos) and Euclidean
Distance (euc) Vector space models are widely used in IR to assess relevance of documents to queries. From simple bag-of-words models to transformer based models, there are several state of the art techniques and pre-trained models to embed the sentences into a vector (Wang et al., 2020). Multilingual  (Sanh et al., 2020) have large scale pre trained models that encode semantic and contextual information in sentence and have achieved state of the art performance in various NLP tasks. From our survey and initial experimentation we found these models provided a better representation for Hindi sentences over other techniques and we obtain the vector representation of question and candidate sentence using LASER and BERT BASE. To measure correspondence and similarity between these vectors following metrics are used 1. Cosine Similarity is a most popular metric that measures the cosine of the angle between two vectors and its value ranges between 0-1. Has been earlier used for QA similarity (Martinez et al., 2012), document clustering, plagiarism detection, IR (Han et al., 2012), (Metcalf and Casey, 2016).
2. Euclidean distance is another similarity measure based on the premise that every instance Cosine similarity computed between vectors obtained using Multilingual LASER and BERT BASE models are denoted by cos l and cos b respectively while euc l and euc b denote euclidean distances. Figure 3 shows cosine similarity computed for the instances shown in figure 1. Euclidean distance is computed in similar manner.

Word Movers Distance (wmd)
Word movers distance is another tool in IR which enables assessing similarity between two sentences with different words (Kusner et al., 2015). It uses vector embedding of words using word2vec algorithm (Mikolov et al., 2013). We compute word movers distance between embedded words of question and candidate answer sentences , using open source SpaCy's (Honnibal and Montani, 2017) wmd implementation along with pre-trained fastText word embedding for Hindi and English words (Bojanowski et al., 2016).

Karaka Relations (kr)
To select correct answer sentence for the given question, understanding the meaning and context of the question and choosing the candidate sentence with similar relevant context and meaning is important. This necessitates sentence analysis. Karaka relations from grammar written for Sanskrit by Indian grammarian Panini in 7th century BCE, provides such a syntactico-semantic analysis of a sentence. As per Paninian framework, meaning of the sentence is encoded in the words as well the relationship between the lexical items (words) in the sentence. The main action is denoted through the verb in the sentence and the grammatical relations between words are categorized into 1. Karaka relations that identify direct participants in the action carried out in the sentence, example 'doer'-karta, 'destination/goal'-karma, 'instrument'-karana, ' recipient/beneficiary'-sampradaan, 'source'apaadaan, 'location'-adhikarana of the action.
2. Relations other than Karaka that identify words that do not have direct role in action, example -reason.
As highlighted by (Sangal and Chaitanya, 1995), Hindi language has morphological rich system of case-endings and post-positions (together called as vibhaktis), which act as explicit markers to identify the above participatory role a word plays in the sentence. The scheme adopted for Hindi treebanking and dependency parsing of a sentence is based on Paninian theory and the dependency annotations are categorized as Karaka relations and non-karaka relations. To extract these relations we obtain the dependency parse of the question using Stanza (Qi et al., 2020) and extract the following: 1. verb present at the root of parse tree to identify the main action being carried out in sentence 2. universal dependency relations (as shown in table 1) corresponding to the six Karaka relations to identify the direct participants in accomplishing the action 3. question word (example 'who', 'when') to identify specific vibhaktis (case-endings) based on set of handcrafted rules For every candidate answer sentence within both Hindi and English contexts, we check if the action verb and direct participants are present. We check question word from Hindi questions only as vibhaktis are specific to Hindi language. For English, candidate with matching root and relations while for Hindi, the candidate with matching root, relations and expected vibhaktis will be semantically more relevant to answer the question.  (Tandon, 2018) We use the similarity measures either individually or in combinations of each other to obtain the feature representation of a candidate sentence.

Dataset
We choose Hindi from the Indo-Aryan language family for the investigation. Hindi being official language in India, is one of the widely used natural languages including in the cyberspace. Multilingual question answering (MLQA) (Lewis et al., 2019) is a multiway aligned benchmark dataset available in seven languages. MLQA is the only available sufficiently large corpus for Hindi or from Indo Aryan Language family that has 5 thousand extractive QA instances. For comparison purpose we also experiment on English MLQA corpus having over 12 thousand instances. Each instance in the corpus has context, question and answer text. Few contexts have more than one question. For sake of simplicity in experimentation, we restrict to the context length of at the most 12 sentences (around 90% of the entire corpus) so that we have 12 target labels corresponding to the sentence indexes to be predicted in this problem.

Model training & answer sentence prediction
Feature extracted for training the ML models are described in section 3 .

Results
We performed several runs on each of the 184 combinations of our 23 feature sets and 8 ML model trained using the same pre-processed dataset, for each of the languages Hindi and English.
Comparison Yardsticks After comparing sufficiently large number of outcomes, we focused on the best results obtained on each feature set from the chosen 8 ML schemes. We found that MN, XgB and CNN are performing as the top three consistently for all feature sets. Thereafter we proceeded to compare the features sets from Hindi and English on these three ML schemes. Table  2 and 3 shows the best validation accuracy obtained over Hindi and English MLQA. We compared the results of each feature set that didn't include Karaka relation(kr) (second column from accuracy tables) with feature set having kr-based feature as the only additional feature combined (third column from accuracy tables). The results unambiguously establish that the performance of answer selection improves significantly when Karaka relation features are combined with other feature sets. Highest accuracy 64.60% for Hindi is reported when Karaka   For English, highest accuracy 70.98% is obtained when Karaka features are combined with content word overlap (cwo+kr). Further adding cosine similarity and word movers distance features (cos b+wmd+cwo+kr) did not show significant improvement in performance over cwo+kr. Adding euclidean distance and longest common sub-string somewhat degraded the overall performance for both English and Hindi. All these results are presented in the graphs in Figure 5. There are 11 feature sets (without Karaka relations), and the change in accuracy over Karaka relations when augmented by each of them is in the red (for Hindi) and yellow (for English) bars. Whereas the improvement in accuracy for each of those sets when Karaka relations augments them is in the adjoining 11 bars (blue-Hindi, green-English).

Interpretation and Discussion
Our outcomes suggest that Karaka relations can be a significant similarity measure for Hindi. It is evident that Karaka relations enhances accuracies significantly across the board when it augments any feature set, whereas the improvement in accuracy when other feature sets augment Karaka relations (alone) is less or negative.
• Feature Overkill Does Not Help We can see that for single features the improvement that can be attributed to Karaka relations, when augmented, is significant to large. Feature sets without Karaka relations do not improve accuracy with addition of more features. For English, euclidean distance measure seems to contribute much less to accuracy when combined with others, and Karaka relations completely dominates it. In the other direction, adding Karaka relations to a large feature set enhances The feature sets (with and without kr), left-to-right, for each 11-bar group of the same color: 1. cos-laser, 2. euc-laser, 3. cos-bert, 4. euc-bert, 5. wmd, 6. cwo, 7. lcs, 8. laser-cos+cwo, 9. laser-cos+cwo+wmd, 10. bert-cos+cwo, 11. bert-cos+cwo+wmd the performance greatly, that gain has not been seen while the feature set grew without Karaka relations. • Semantic Similarity The similarity measures that do contribute to accuracy enhancement when combined with Karaka relations are expected to capture semantic information indirectly. Karaka relations, when exploited fully, will help capture semantic and context information directly. In our experiments we have not yet attempted that, we have not exploited the full potential of Karaka relations. Therefore, these results suggest that research in this direction -exploiting Karaka relations fully to capture semantic information directly -can possibly reduce computational overheads in featureheavy models by eliminating other features.
In this work, we investigate the influence of various similarity based feature sets on answer sentence selection task in Hindi QA, and outcomes highlight that Karaka relations can be a significant similarity measure. We have not exploited the full potential of using Karaka relations to obtain the syntactico-semantic sentence analysis for measuring question and answer candidate similarity. Doing so is not computationally intensive, but the morphological analysis, disambiguation, lexical analysis, and different parsing actions that are needed to go the whole hog requires extensive linguistic knowledge converted into computational models. If this is sought to be done using ML again, then choosing or curating relevant datasets with sufficient variety of examples including ambiguity and other linguistic challenges will need, in turn, the same extended linguistics expertise. Our experimentation and visualisation of the outcomes surely makes a strong case to attract this investment into this enterprise. Precisely, we demonstrate that such linguistic knowledge, incorporated into Karaka relations extraction and use of the same for similarity measures in the way we did, can reduce computational costs for the same accuracy goals and the similar datasets. Moreover, refinement of Karaka relations extraction to optimal quality will definitely reduce the need for availability of large pre-trained language models.