{'fraction_non_alphanumeric': 0.0288052943225357, 'fraction_numerical': 0.007802159526297457, 'mean_word_length': 4.983951646519383, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DRCoVe: An Augmented Word Representation Approach using Distributional and Relational Context

Word representation using the distributional information of words from a sizeable corpus is considered efficacious in many natural language processing and text mining applications. However, distributional representation of a word is unable to capture distant relational knowledge, representing the relational semantics. In this paper, we propose a novel word representation approach using distributional and relational contexts, DRCoVe, which augments the distributional representation of a word using the relational semantics extracted as syntactic and semantic association among entities from the underlying corpus. Unlike existing approaches that use external knowledge bases representing the relational semantics for enhanced word representation, DRCoVe uses typed dependencies (aka syntactic dependencies) to extract relational knowledge from the underlying corpus. The proposed approach is applied over a biomedical text corpus to learn word representation and compared with GloVe, which is one of the most popular word embedding approaches. The evaluation results on various benchmark datasets for word similarity and word categorization tasks demonstrate the effectiveness of DRCoVe over the GloVe.


Introduction
Understanding contextual semantics of words is crucial in many natural language processing (NLP) applications. Recent trends in text mining and NLP suggest immense interest towards learning word embedding or word representation in a vector space from a large corpus, which could be useful for a variety of applications like text classification (Lai et al., 2015), clustering (Wang et al., 2015), and sentiment analysis (Tang et al., 2014). In addition, researchers are devising methods to learn phrase-, sentence-, or document-level embeddings for various NLP applications. Word embeddings capture implicit semantics and hence attracted many researchers to explore and exploit a tremendous amount of available unstructured corpora for efficient word representation by employing mainly unsupervised learning approaches. Further, the growth and availability of domain-specific massive text corpora can be exploited to learn domain-specific word representation.
Although different approaches for learning word embeddings have been proposed in the prior works, they are mostly based on distributional representation of words, considering the neighbors of a word within a fixed context window. These algorithms map sparse representation of words to a lower dimensional vector space where words with similar context appear nearby each other. However, distributional representation of words learned by these algorithms suffer from two important limitations -(i) unable to capture the relational semantics of rare co-occurring words within the corpus, and (ii) unable to capture the relational semantics of words that are outside the purview of the context window. The first limitation is that a large corpus, though represents different contextual information, may have rare co-occurrence of two words because it might not be large enough to possess sufficient count of the co-occurrence of semantically similar word pairs.
To overcome this limitation, researchers have incorporated knowledge into these distributional word representations from external knowledge bases (KBs). In this direction, semantically related words in terms of relations like synonymy, hypernymy, and meronymy from KBs like WordNet (Miller, 1995), Freebase (Bollacker et al., 2008) have been used to learn better representation of words (Alsuhaibani et al.,Figure 1: An exemplar dependency parse tree generated by the Stanford parser using DependenSee 3.7.0 2018; Celikyilmaz et al., 2015). This makes these approaches dependent upon the external KBs to enhance the efficacy of word representation. Although KBs provide significant information about word relations, they are scanty with limited entries for each word and does not represent any contextual information. In addition, since KBs are manually curated and maintained, they are not comprehensive.
The second limitation is that the distributional word representations are unable to capture the relational semantics of words due to their dependence on the fixed context window, and hence ignore the semantic associations between words that are outside the purview of the context window. For example, in the sentence, "cholera is an infectious disease characterized by watery diarrhea, vomiting, severe dehydration, and muscle cramps", the word pairs (cholera, dehydration) and (cholera, cramps) have long-range dependency. However, both dehydration and cramps are semantically associated with cholera as they are its symptoms. In case of fixed context window size, e.g. 5, such long range dependency relationships will not be captured. Further, if we increase the size of the context window, it will adversely impact the embedding representation due to the inclusion of irrelevant and weak contextual words. Additionally, in case of domain-specific corpus for learning word embedding, the semantic relation between cholera and dehydration, or cholera and cramps would be very vital because dehydration and muscle cramps are the symptoms of cholera. These relational semantics can be captured by dependency grammar that shows syntactic and semantic relationships between words of a sentence. To this end, Levy and Goldberg (2014a) presented a dependency-based word representation learning approach to incorporate the syntactic contexts instead of linear contexts. However, existing literatures have no approach that learn word representation using syntactic contexts extracted from inter-relationships of words based on the dependency tuples generated by the language-parser. For example, in figure 1, the syntactic contexts using only the head and modifier words of the dependency tuples generated by the parser shows direct dependency relation between cholera and disease through nsubj dependency relation; but, it doesn't not show any relational semantics between cholera and watery, diarrhea, vomiting, dehydration, and cramps as they are not directly linked to cholera by any dependency relations. Therefore, extraction of such relations to augment word representations would be very helpful for various domain-specific NLP tasks such as classification of disease-related documents or texts. To the best of our knowledge, in the existing literatures, no such approach exists that utilizes the relational semantics extracted from a large corpus to enhance the distributional representation of words.
In this paper, we present an augmented approach, DRCoVe, to use both text corpus and an extracted repository of semantically related triplets from the corpus to learn efficient word representation. The proposed approach first initializes the word representation to low-dimensional real-valued vectors generated from the singular value decomposition (SVD) of positive pointwise mutual information (PPMI) matrix of the underlying corpus and the relational semantic repository. The initial word vectors from the corpus are augmented using vectors from the relational semantic repository, provided the words from the corpus occur in the vocabulary of the relational semantic repository. In the proposed approach, we implement a modified GloVe (Pennington et al., 2014) objective function for cost optimization to incorporate vector representations from the relational knowledge repository with the initial vectors from the corpus. In brief, the main contributions of this paper can be summarized as follows.
• We propose DRCoVe, a novel approach of learning and augmentation of word representation from a corpus that can handle both long-and short-range dependencies among words.
• The model combines the benefits of point-wise mutual information, singular value decomposition, and neural network-based updation.
• Compared to existing approaches, the proposed model performs considerably better on different benchmark datasets.
Rest of the paper is organized as follows. Section 2 presents a brief review of the existing works on learning word representations. Section 3 presents background details of the concepts used in this paper. Section 4 presents the detailed description of the proposed model. Section 5 presents the experimental details and evaluation results. Finally, section 6 concludes the paper and provides future directions of research.

Related Works
Recently, a number of different learning algorithms have been proposed to learn the low-dimensional dense representation of words generally called word embedding used in different NLP tasks such as named entity recognition (Collobert et al., 2011), sentiment analysis (Tang et al., 2014). In this regard, two popular word representation models: continuous bag of words (CBOW) and skip gram (SG) (Mikolov et al., 2013a) models based on neural networks have gained momentum in learning distributed word representation by exploiting the local context of words co-occurring within a given context window. The CBOW predicts the target word given the surrounding context words while SG predicts the surrounding context words given the current word. Similarly, GloVe (Pennington et al., 2014) is another popular method of learning word representation based on global co-occurrence matrix that predicts global co-occurrence between target and context words by employing randomly initialized vectors of desired dimensions. These models learn embeddings only from the corpus without incorporation of any external knowledge. However, in this direction, numerous studies (Yu and Dredze, 2014;Xu et al., 2014;Alsuhaibani et al., 2018) have attempted to incorporate the relational information from KBs for word representation. In Yu and Dredze (2014), the authors proposed an approach to jointly learn embeddings from a corpus and a similarity lexicon (synonymy) by assigning high probabilities to words that appear in the similarity lexicon using joint objective functions of relation constraint models (RCM) and CBOW. Similarly, Xu et al. (2014) used the relational and categorical information as regularization parameters to the SG training objective function to improve the word representation. The CBOW based models normalize target word probabilities for the whole vocabulary, hence, computationally very expensive for large corpora.
In Ghosh et al. (2016), the authors proposed vocabulary driven skip-gram with negative sampling (SGNS) to learn disease-specific word vectors from health-related news corpus by incorporating disease-related vocabulary. Most of the proposed word representation approaches are based on either of the two models (CBOW or SG), or their variants (SGNS, SGHS) of Word2Vec algorithm either by linearly combining additional objective functions or adding as regularizers. Alsuhaibani et al. (2018) used WordNet to extract eight different types of relations such as synonymy, antonymy, hypernymy, meronymy, and so on to learn joint embeddings. They used a linear combination of GloVe and KB-based objective functions.
All the discussed and existing approaches ignore the relational semantics between the words, which are outside the purview of context-window.

Background and Problem Definition
This section presents the notations and the background details of the important concepts used in the proposed approach.
Notations: Suppose a corpus C has n number of documents d 1 , d 2 , . . . , d n , and D represents the collection of target and context words pairs (w, c) obtained from C for a given context window size l, where context words of a target word w i are the surrounding words w i−l , . . . , w i−1 , w i+1 , . . . , w i+l .
In addition, assume that V w and V c represent word and context vocabularies respectively for corpus D. We assume that n (w,c) represents the total count of (w, c) pair in D such that the target word w and context word c appear together within the context window l, n w and n c denote the occurrence of w and c respectively in D such that n w = ĉ∈Vc n (w,ĉ) and n c = ŵ∈Vw n (ŵ,c) . The association between every pair of target and context words of V w and V c is presented in a matrix M such that each row of the matrix represents the vector of a target word w ∈ V w and each column represents vector of a context word c ∈ V c and every element M i,j represents the association between the i th target word w i and j th context word c j . Further, assume a relational semantic repository R l consisting of all the relational semantic triplets extracted from the corpus C. In addition, assume V represents the vocabulary of R l . In the paper, alphabets w and c in bold typeface represent vectors.
GloVe: It is a neural network-based machine learning algorithm to learn an efficient lower dimensional dense representation of words in an embedding space. It uses global co-occurrence matrix to learn distributed representation of words from a text corpus. Initially, it creates co-occurrence matrix M with rows representing target words for which we want to learn word representation and the columns represent the context words co-occurring with the target words in the corpus within a given context window. In M , each entry, say, M i,j represents the sum of the reciprocal of the distance of co-occurring target and context words. GloVe implements weighted least square regression objective function to minimize the loss J g as given in equation 1, where f (M w,c ) is the weight function to find weight between a target word w and context word c as given in equation 2, and b w and b c are the bias terms for the underlying target and context words respectively. In the equation 2, α = 0.75 is a hyper-parameter and x max = 100. The objective of GloVe is to minimize the squared difference between the inner product of word and context vectors w and c, and the logarithm of their co-occurrence count in D. (2) In GloVe, learning process starts by assigning random vectors of desired dimensions to the target and context words and then updating them during the learning process with an objective to reduce the weighted least square loss as given in equation 1.
Pointwise Mutual Information: In the existing literature, researchers have used different metrics such as co-occurrence count in GloVe to represent the association between a word and context pair (w, c). However, simple frequency count is not the best measure of association as it does not incorporate any contextual information. The pointwise mutual information (PMI) is another measure of association and better as compared to co-occurrence count. It measures how often two events co-occur compared to what we would expect if they were independent as defined in equation 3 (Jurafsky and Martin, 2018). There can be target and context word pairs (w ∈ V w and c ∈ V c ) which do not appear together within the given context window l in the corpus and for such pairs n (w,c) = 0, and therefore P M I(w, c) = log(0) = −∞. To avoid this situation, positive pointwise mutual information (PPMI) has been used in which negative PMI values are mapped to zero as given in equation 4. In addition, Bullinaria and Levy (2007) showed that PPMI performs better than PMI in finding semantic similarity. PPMI measures are widely used to find semantic similarity, however, these matrices are highly sparse and need huge computational resources. One measure is to convert such sparse vectors into low dimensional dense vectors to improve computational efficiency and generalization. In this regard, dimensionality reduction is a way to find low dimensional dense vectors using matrix factorization techniques such as SVD.
Singular Value Decomposition: It is a dimensionality reduction method which decomposes a symmetric matrix M m×n into three matrices U , Σ, and V such that M = U ·Σ·V . The matrices U and V are orthogonal matrices while Σ is a diagonal matrix of singular values. To obtain d dimensional vectors, the matrix M is decomposed to U m×d , Σ d×d , and V d×n corresponding to top d singular values. The d-dimensional rows of matrix W = U · √ Σ are dense vectors which are the approximate representative of high dimensional rows of M . The matrix W is considered as a dense vector representation of words, while the matrix C = V T · √ Σ can be considered as context representation. The matrices W and C thus obtained are used as initial word and context representations respectively. These resulting representations need to fulfill minimization of error in matrix decomposition.

Proposed Approach
This section presents the detailed description of the proposed approach, starting from the mechanism to generate initial word representation from the corpus, their augmentation through relational semantics, and finally, adaptive updation of word vectors. A detailed description of each step of the proposed approach is presented in the following subsections.

Initial Vector Representation
To learn word representation of desired dimension, we first need to initialize the vectors for each target and context words pair of the corpus that can be further augmented using their relational semantics and updated based on the weighted least square loss minimization process. Before neural-based approaches, distributed word representations were based on count-based vectors such as tf-idf and SVD-based vectors. Recent advancements in neural network-based word representation have shown significant improvement in its performance in various NLP tasks. The neural network-based word representations are based on prediction (Mikolov et al., 2013b,a) of either the target word given the context within the specified context window or vice versa. However, recent studies (Levy and Goldberg, 2014b;Levy et al., 2015) have shown that the neural network-based embedding learned using Word2Vec or GloVe models are comparable in performance with the traditional representation of vectors obtained through the decomposition of PPMI matrix. Therefore, to incorporate the benefits of traditional decomposition-based vectors, the proposed approach generates initial word representation using vectors obtained from SVD-based factorization of PPMI matrix. To this end, we first create a co-occurrence matrix M considering the co-occurrence count of every (w, c) pair of target and context words from V w and V c respectively that is further mapped to a PPMI matrix M p . Thereafter, the M p is factorized using SVD to generate initial low dimensional dense vector representations of target and context words as W = U · √ Σ and C = V T · √ Σ, respectively from the corpus that incorporate the distributional semantics. Similarly, the same process is repeated for relational semantic repository R l to generate the initial vector representation of target and context words asŴ = U · √ Σ andĈ = V T · √ Σ, respectively from R l . The initial vectors of target and context words from the corpus are further augmented using the vectors generated from the relational semantic repository. A detailed description of the augmentation process is described in the following section.

Objective Function Augmentation
To minimize the decomposition error we followed the GloVe approach of optimization of the initial representation of vectors. GloVe method learns continuous word representation from a corpus using the global co-occurrence matrix. However, Glove does not incorporate any additional or domain-specific knowledge and suffers from two important limitations as discussed in section 1. Therefore, during optimization we performed the augmentation of initial word representation from the corpus by merging with the initial word representation from the relational semantic repository. To augment the additional information during learning, we define an augmented objective function J a similar to GloVe as given in equation 5, where f (p w,c ) is the weight function to assign weight between every pair (w, c) of target and context words as given in equation 6, and b w and b c are the bias values for w and c respectively, and p w,c is the PPMI value between w and c. In equation 6, α is a hyper parameter and we used 0.75 as its value as used in GloVe.
, 1 ™ (6) Thereafter, we employed the relational semantics from the extracted relational semantic repository R l consisting of vocabulary V to augment the learning process. The input corpus C consist of target and context words pairs (w, c) ∈ D. Based on V, we grouped the (w, c) pairs of D into three categories -D ∧ , D ∼ , and D ⊕ such that the target and context words belongs to V neither target nor the context word belongs to V i.e either the target or the context word belongs to V We need to consider each of these (w, c) pair categories especially while merging to generate augmented word representation.
In case of D ∧ , as both the target and context words belong to V, we considered the merged vectors from the corpus and the relational semantic repository corresponding to target and context words such that w = 0.5 * (w +ŵ) and c = 0.5 * (c +ĉ), where w and c are the initial vectors from corpus andŵ andĉ are the initial vectors from relational semantic repository. For category D ∼ , we considered the initial vectors from the corpus only as neither of the two words belongs to V, hence, we have w = w and c = c. Similarly, in case of D ⊕ , as either of the two words belongs to V but not both, we took the merged vector for target or context word depending upon which word belongs to V. In this case, if target word belongs to V, we take w = 0.5 * (w +ŵ) and if context word belongs to V, we consider c = 0.5 * (c +ĉ).

Adaptive Updation of Parameters
We performed the parameter updation during learning process based on a well-known gradient descent technique called AdaGrad (Duchi et al.,  δJ δw =g t,w = c∈Vc f (pw,c)(w T ·c +bw+bc−log(pw,c))·c (7) δJ δc =g t,c = w∈Vw f (pw,c)(w T ·c +bw+bc−log(pw,c))·w AdaGrad algorithm is suitable for dealing with sparse data as it performs larger updates for infrequent words, and smaller updates for frequent words. The update equation is shown as follows: where, w is the a merged target word vector, g t,w is the gradient at time t, and g 2 τ,w is the squared gradient at time τ for the target word vector w . Similarly, updates for context word and biases are performed according to the following equations.

Experimental Setup and Results
The DRCoVe is evaluated on different benchmark datasets using two evaluation tasks -word similarity and concept categorization. This section presents a brief description of corpus and relational semantic repository used in the evaluation process, experimental setup, and finally presents the evaluation results.

Corpus and Relational Semantic Repository
The DRCoVe is evaluated on a biomedical text corpus crawled from PubMed 1 , an online repository of millions of citations and abstracts related to biomedicine, health, life and behavioral sciences, and bioengineering. The abstracts are the source of rich information related to diseases, symptoms, pathogens, vectors, and their transmission and etiologies. PubMed provides access to the abstracts of documents through axis 2.1.6.2 API 2 . The crawled corpus C consist of 16,337 PubMed documents related to four diseases -cholera, dengue, influenza, and malaria. In addition, a relational semantic repository R l is created by extracting relational triplets < arg 1 , relation, arg 2 > based on typed dependencies generated by Stanford parser 3 that are filtered using MetaMap 4 to identify meaningful disease-symptom triplets. The repository R l is used to augment the learning process of word representation. We have extracted the association between the diseases and symptoms using the 1 https://www.ncbi.nlm.nih.gov/pubmed/ 2 http://axis.apache.org/axis2/java/core/ 3 http://nlp.stanford.edu/software/lex-parser.shtm 4 https://metamap.nlm.nih.gov/ approach defined in (Parwez et al., 2018;Abulaish et al., 2019).

Experimental Setup
The documents of the corpus C are tokenized and processed by removing numbers, punctuations, and stop words. We experimented with the context window size l of 5 and 10 (i.e. for l = 5, the context words are the 5 preceding and 5 succeeding words to the target word) to extract the context words from the corpus. The co-occurrence matrix is created using the co-occurrence frequencies of the target and context words pair within the corpus. The co-occurrence matrix is further mapped into PPMI matrix, which is further factorized using SVD to get the initial word vector of desired dimension d ∈ {100, 200}. A similar procedure is repeated for relational semantic repository R l and initial vectors are generated for the target and context words. Thereafter, initial word representation of corpus is augmented using the word representation of relational semantic repository which is then optimized using the objective function defined in equation 5. We used a stochastic gradient-based algorithm AdaGrad with the learning rate η = 0.05 for optimization. The proposed algorithm is executed for 50 iterations to converge into an optimum solution. As a result, we obtain two sets of enhanced embeddings, one for the target words of vocabulary V w and another for the context words of vocabulary V c . It has been shown that when the two embeddings of a word are combined by taking an average of the corresponding word vectors, the resultant embedding performs better (Pennington et al., 2014). We have presented results for both the word and context representation in addition to their merged representation.

Evaluation Results and Comparative Analysis
The quality of the learned word vectors based on DRCoVe is evaluated using concept categorization and similarity prediction tasks. Concept Categorization: We evaluated the quality of learned word embedding based on concept categorization. It is the grouping of concepts from a given set of concepts into different categories. It evaluates the word representation by clustering the learned vectors into different groups. The performance is assessed based on the extent to which each cluster possesses concepts from a given category. The evaluation metric is  Word Similarity: To evaluate learned vectors on word similarity task, we computed cosine similarity between learned embedding of word pairs and evaluated it based on average similarity rating assigned by human annotators to these word pairs from the benchmark datasets. The idea here is that the learned embeddings encapsulate semantics of the words if there is greater extent of correlation between the similarity score computed from the learned word vectors and the similarity score assigned by the human annotators. We calculated Spearman's rank correlation coefficient between the cosine similarity of learned embeddings and human rated similarity of word pairs. We used 9 different benchmark datasets -MTurk, RG65, RW, SCWS, SimLex999, TR9856, WS353, WS353R, and WS353S for evaluation. In addition, we also compared the quality of learned representation in terms of similarity task with the two variants of GloVe: GloVe W and GloVe Merged. The evaluation and comparison results on different combination of context window size and dimensionality on the benchmark datasets for word similarity are given in tables 5, 6, 7, and 8 respectively. On analysis of tables, it can be found that the context and merged vectors of DRCoVe are significantly better as compared to GloVe word vectors and merged vectors except RW, where GloVe is better.

Conclusion and Future Direction
Word embeddings learned from diverse sources using methods like GloVe as the distributional representation of words have been employed to resolve numerous natural language processing problems with considerable accuracy. However, these distributional representations are unable to capture the relational semantics of distant words and the words with rare co-occurrences in the corpus. In this paper, we have proposed DRCoVe, an augmentation approach of distributional word representations from a corpus with relational semantic information extracted from the corpus to learn enhanced word representation. We compared the proposed model based on semantic similarity and concept categorization tasks on different benchmark datasets and found that the word representation learned by DRCoVe shows better performance than the GloVe model in most of the datasets. The learned word representations could be useful for various NLP tasks like text classification or concept categorization. Learning word representations over much larger corpus and evaluation of their efficacy for short texts