{'fraction_non_alphanumeric': 0.0427007299270073, 'fraction_numerical': 0.022141119221411192, 'mean_word_length': 4.651770367823994, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Lexical Semantics for Sentential Semantics: Modeling Selectional Preference and Similar Words in a Latent Variable Model

Sentence Similarity [SS] computes a similarity score between two sentences. The SS task differs from document level semantics tasks in that it features the sparsity of words in a data unit, i.e. a sentence. Accordingly it is crucial to robustly model each word in a sentence to capture the complete semantic picture of the sentence. In this paper, we hypothesize that by better modeling lexical semantics we can obtain better sentential semantics. We incorporate both corpus-based (selectional preference information) and knowledge-based (similar words extracted in a dictionary) lexical semantics into a latent variable model. The experiments show state-of-the-art performance among unsupervised systems on two SS datasets.


Introduction
Sentence Similarity [SS] is emerging as a crucial step in many NLP tasks that focus on sentence level semantics such as word sense disambiguation (Guo and Diab, 2010;Guo and Diab, 2012a), summarization (Zhou et al., 2006), text coherence (Lapata and Barzilay, 2005), tweet clustering (Sankaranarayanan et al., 2009;Jin et al., 2011), etc. SS operates in a very small context, on average 11 words per sentence in Semeval-2012 dataset (Agirre et al., 2012), resulting in inadequate evidence to generalize to robust sentential semantics.
Weighted Textual Matrix Factorization [WTMF] (Guo and Diab, 2012b) is a latent variable model that outperforms Latent Semantic Analysis [LSA] (Deerwester et al., 1990) and Latent Dirichelet Allocation [LDA] (Blei et al., 2003) models by a large margin in the SS task, yielding state-of-the-art performance on the LI06 (Li et al., 2006) SS dataset. However, all of these models make harsh simplifying assumptions on how a token is generated: (1) in LSA/WTMF, a token is generated by the inner product of the word latent vector and the document latent vector; (2) in LDA, all the tokens in a document are sampled from the same document level topic distribution. Under this framework, they ignore rich linguistic phenomena such as inter-word dependency, semantic scope of words, etc. This is a result of simply using document IDs as features to represent a word.
Modeling quality lexical semantics in latent variable models does not draw enough attention in the community, since people usually apply dimension reduction techniques for documents, which have abundant words for extracting the document level semantics. However, in the SS setting, it is crucial to make good use of each word, given the limited number of words in a sentence. We believe a reasonable word generation story will avoid introducing noise in sentential semantics, encouraging robust lexical semantics which can further boost the sentential semantics. In this paper, we explicitly encode lexical semantics, both corpus-based and knowledge-based information, in the WTMF model, by which we are able to achieve even better results in SS task.
The additional corpus-based information we exploit is selectional preference semantics (Resnik, 1997), a feature already existing in the data yet ignored by most latent variable models. Selectional preference focuses on the admissible arguments for a word, thus capturing more nuanced semantics than the sentence IDs (when applied to a corpus of sentences as opposed to documents). Consider the following example: In WTMF/LSA/LDA, a word will receive semantics from all the other words in a sentence, hence, the word oil, in the above example, will be assigned the incorrect finance topic that reflects the sentence level semantics. Moreover, the problem worsens for adjectives, adverbs and verbs, which have a much narrower semantic scope than the whole sentence. For example, the verb say should only be associated with analyst (only receiving semantics from analyst), as it is not related to other words in the sentence. In contrast, oil, according to its selectional preference, should be associated with crude indicating the resource topic. We believe modeling selectional preference capturing local evidence completes the semantic picture for words, hence further rendering better sentential semantics. To our best knowledge, this is the first work to model selectional preference for sentence/document semantics.
We also integrate knowledge-based semantics in the WTMF framework. Knowledge-based semantics, a human-annotated clean resource, is an important complement to corpus-based noisy cooccurrence information. We extract similar word pairs from Wordnet (Fellbaum, 1998). Leveraging these pairs, an infrequent word such as purchase can exploit robust latent vectors from its synonyms such as buy. Similar words pairs can be seamlessly modeled in WTMF, since in the matrix factorization framework a latent vector profile is explicitly created for each word, while in LDA all the data structures are designed for documents/sentences. We construct a graph to connect words according to the extracted similar word pairs, to encourage similar words to share similar latent vector profiles. We will refer to our proposed novel model as WTMF+PK.

Weighted Textual Matrix Factorization
Our previous work (Guo and Diab, 2012b) models the sentences in the weighted matrix factorization framework ( Figure 1). The corpus is stored in an M × N matrix X, with each cell containing the TF-IDF values of words. The rows of X are M distinct words and columns are N sentences. As in Figure  1, X is approximated by the product of a K × M matrix P and a K × N matrix Q. Accordingly, each sentence s j is represented by a K dimensional latent vector Q ·,j . Similarly a word w i is generalized by P ·,i . P and Q is optimized by minimize the objective function: where λ is a regularization term. Missing tokens are modeled by assigning a different weight w m for each 0 cell in the matrix X. We can see the inner product of a word vector P ·,i and a sentence vector Q ·,j is used to approximate the cell X ij .
The graphical model of WTMF is illustrated in Figure 2a. A w i /s j node is a latent vector P ·,i /Q ·,j , corresponding to a word/sentence, respectively. A shaded node is a non-zero cell in X, representing an observed token in a sentence. For simplicity, the missing tokens and weights are not shown in the graph.

Corpus-based Semantics: Selectional Preference
In this paper, we focus on selectional preference that reflects the association of two words: if two words form a bigram, then the two words should share similar latent dimensions. In the previous example, crude and oil form a bigram, and they share the resource topic. In our framework, this is implemented by adding extra columns in X, so that each additional column corresponds to a bigram, treating each bigram as a pseudo-sentence for the two words. The graphical model is illustrated in Figure 2b. Therefore, oil will receive more resource topic from crude through the bigram crude oil, instead of only finance topic from the sentence as a whole.
Each non-zero cell in the new columns of X, i.e. an observed token in a bigram (pseudo-sentence), is given a different weight: Figure 2: WTMF+PK model (WTMF + corpus-based Selectional [P]references semantics + [K]nowledge-based semantics): a w/s/b node represents a word/sentence/bigram, respectively f req(j) denotes the frequency of bigram j appearing in the corpus, hence the strength of association is differentiated such that higher weights are assigned on the more probable bigrams. The coefficient γ is the importance of selectional preference. A larger γ indicates that we trust the selectional preference over the global sentential semantics.

Knowledge-based Semantics: Similar Word Pairs
We first extract synonym pairs from WordNet, which are words associated with the same sense, synset. We further expand the set by exploiting the relations defined in WordNet. For the extracted words, we consider the first sense of each word, and if it is connected to other senses by any of the WordNet defined relations (hypernym, similar words, etc.), then we treat the words associated with the other senses as similar words. In total, we are able to discover 80K pairs of similar words for the 46K distinct words in our corpus. Given a pair of similar words w i 1 /w i 2 , we want the two corresponding latent vectors P ·,i 1 /P ·,i 2 to be as close as possible, namely the cosine similarity to be close to 1. Accordingly, a term is added in equation 1 for each similar word pair w i 1 /w i 2 : (2) |P ·,i | denotes the length of the vector P ·,i . The coefficient δ, analogous to γ, denotes the importance of the knowledge-based evidence. The Figure 2c shows the final WTMF+PK model.

Inference
In (Guo and Diab, 2012b) we use Alternating Least Square [ALS] for inference, which is to set the derivative of equation 1 for P/Q to 0 and iteratively compute P/Q by fixing the other matrix (Srebro and Jaakkola, 2003). However, it is no longer applicable with the new term (equation 2) involving the length of word vectors |P ·,i |. Therefore we approximate the objective function by treating the vector length |P ·,i | as fixed values during the ALS iterations: where P ·,s(i) are the latent vectors of similar words of word i; the length of these vectors in the current iteration are stored in L s(i) (similarly L i is the current length of P ·,i ) (cf. (Steck, 2010;Guo and Diab, 2012b) for optimization details).

Experimental Setting
We build the model WTMF+PK on the same corpora as used in our previous work (Guo and Diab, 2012b), comprising the following: Brown corpus (each sentence is treated as a document), sense definitions from Wiktionary and Wordnet (only definitions without target words and usage examples). We follow the preprocessing steps in (Guo and Diab, 2012c): tokenization, pos-tagging, lemmatization and further merge lemmas. The corpus is used for building matrix X.
The evaluation datasets are LI06 dataset and Semeval-2012 STS [STS12] (Agirre et al., 2012) dataset. LI06 consists of 30 sentence pairs (dictionary definitions). For STS12, 1 the training data (2000 pairs) are used as the tuning set for setting the parameters of our models. This data comprises msrpar, msr-vid, smt-eur. Once the models are tuned, we evaluate them on the STS12 test data that comprises 3150 sentence pairs from msr-par, msr-vid, smt-eur, smt-news, On-WN. It is worth noting that smt-news and On-WN are not part of the tuning data. We use cosine similarity to measure the similarity scores between two sentences. Pearson correlation between the system's answer and gold standard similarity scores is used as the evaluation metric.
We include three baselines LSA, LDA and WTMF using the setting described in (Guo and Diab, 2012b). We run Gibbs Sampling based LDA for 2000 iterations and average the model over the last 10 iterations. For WTMF, we run 20 iterations and fix the missing words weight at w m = 0.01 with a regularization coefficient set at λ = 20, which is the best condition found in (Guo and Diab, 2012b). Table 1 summarizes the results at dimension K = 100 (the dimension of latent vectors). To remove randomness, each reported number is the averaged results of 10 runs. Based on the STS tuning set, we experiment with different values for the selectional preference weight (γ = {0, 1, 2}), and likewise for the similar word pairs weight varying the δ value as follows δ = {0, 0.1, 0.3, 0.5, 0.7}. The performance on STS12 tuning and test dataset as well as on the LI06 dataset are illustrated in Figures 3a, 3b and 3d. The parameters of model 6 in Table 1 (γ = 2, δ = 0.3) are the chosen values based on tuning set performance. Table 1 shows WTMF is already a very strong baseline: it outperforms LSA and LDA by a large margin. Same as in (Guo and Diab, 2012b), LSA performance degrades dramatically when trained on a corpus of sentence sized documents, yielding results worse than the surface words baseline 31% (Agirre et al., 2012). Using corpus-based selectional preference semantics alone (model 4 WTMF+P in Table  1) boosts the performance of WTMF by +1.17% on the test set, while using knowledge-based semantics alone (model 5 WTMF+K) improves the over the WTMF results by an absolute +2.31%. Combining them (model 6 WTMF+PK) yields the best results, with an absolute increase of +3.39%, which suggests that the two sources of semantic evidence are useful, but more importantly, they are complementary for each other. Table 1 also presents the performance on each individual dataset. The gain on each individual source is not as much as the overall gain, which suggests part of the overall gain comes from the correct ranking of intra-source pairs. Note that WTMF+PK improves all individual datasets except smt-eur. This may be caused by too many overlapping words in the sentence pairs in smt-eur, while our approach focuses on extracting similarity between different words.

Evaluation on the STS12 datasets
Observing the performance using different values of weights in figure 3a and 3b, we can conclude that the selectional preference and similar word pairs yield very promising results. The trends hold in different parameter conditions with a consistent improvement. Figure 3c illustrates the impact of dimension K = {50, 75, 100, 125, 150} on WTMF and WTMF+PK. Generally a larger K leads to a higher Pearson correlation, but the improvement is tiny when K ≥ 100 (0.1% increase).
Compared to all the unsupervised systems that participated in Semeval STS 2012 task, WTMF+PK yields state-of-the-art performance (70.70%). 2 In (Guo and Diab, 2012c) we also apply WTMF (K = 100) on STS12, achieving a correlation of 69.5%. However, additional data is incorporated in the training corpora: (1) STS12 tuning set; (2) for WordNet and Wiktionary data, the target words are also included in the definitions (hence synonym pairs were used); (3) the usage examples of target words were also appended to the definitions. 3 While trained with this experimental setting, our model WTMF+PK (γ = 2, δ = 0.3, K = 100) is able to reach an even higher correlation of 72.0%.    Figure 3d presents the results obtained on the LI06 data set at different weight values for the corpusbased selectional preference semantics γ and for the knowledge-based semantics δ. Our previous experiments (Guo and Diab, 2012b) show that WTMF is the state-of-the-art model on LI06. With lexical semantics explicitly modeled, WTMF+PK yields better results than WTMF (see Table 1). It should be noted that LI06 prefers a smaller similar word pair weight ( a δ = 0.1 yields the best performance around of 90.75%), yet in almost all conditions WTMF+PK outperforms WTMF as shown in Figure 3d.

Related Work
SS has progressed immensely in recent years, especially with the establishment of the Semantic Textual Similarity task in SEMEVAL 2012. Early work in SS focused on word pair similarity in the high dimensional space (Li et al., 2006;Liu et al., 2007;Islam and Inkpen, 2008;Tsatsaronis et al., 2010;Ho et al., 2010), where co-occurrence information was not efficiently exploited. Researchers (O'Shea et al., 2008) find LSA does not yield good performance. In (Guo and Diab, 2012b;Guo and Diab, 2012c), we show the superiority of the latent space approach in WTMF. In this paper, we improve the WTMF model and achieve state-of-the-art Pearson correlation on two standard SS datasets. There are latent variable models designed for lexical semantics, such as word senses (Boyd-Graber et al., 2007;Guo and Diab, 2011), function words (Griffiths et al., 2005), selectional preference (Ritter et al., 2010), synonyms and antonyms (Yih et al., 2012), etc. However little improvement is shown on document/sentence level semantics: (Ritter et al., 2010) and (Yih et al., 2012) focus on selectional preference and antonym identification, respectively; in (Griffiths et al., 2005) the LDA performance degrades in the text categorization task including the modeling of function words. Rather, we concentrate on nuanced lexical semantics phenomena that could benefit sentential semantics.

Conclusion
We incorporate corpus-based (selectional preference) and knowledge-based (similar word pairs) lexical semantics into a latent variable model. Our system yields state-of-the-art unsupervised performance on two most popular and standard SS datasets.

Acknowledgment
This work is supported by the IARPA SCIL program.