{'fraction_non_alphanumeric': 0.03277545327754533, 'fraction_numerical': 0.012761506276150627, 'mean_word_length': 4.832045546970313, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hyperspherical Query Likelihood Models with Word Embeddings

This paper presents an initial study on hyperspherical query likelihood models (QLMs) for information retrieval (IR). Our motivation is to naturally utilize pre-trained word embeddings for probabilistic IR. To this end, key idea is to directly leverage the word embeddings as random variables for directional probabilistic models based on von Mises-Fisher distributions which are familiar to cosine distances. The proposed method enables us to theoretically take semantic similarities between document and target queries into consideration without introducing heuristic expansion techniques. In addition, this paper reveals relationships between hyperspherical QLMs and conventional QLMs. Experiments show document retrieval evaluation results in which a hyperspherical QLM is compared to conventional QLMs and document distance metrics using word or document embeddings.


Introduction
In information retrieval (IR), language modeling is known to be one of the most successful techniques (Ponte and Croft, 1998). A typical usage is query likelihood models (QLMs), in which language models are constructed from each retrieved document. In QLM-based probabilistic IR, the documents are ranked by probabilities for which a query can be generated by the document language model.
In this field, categorical QLMs which model generative probability of words using categorical distributions are fundamental models (Ponte and Croft, 1998;Zhai and Lafferty, 2001). It is known that the categorical QLMs do not perform well for vocabulary mismatches because categorical distribution cannot consider semantic relationships between words. Therefore, several expansion techniques such as query expansion (Bai et al., 2005), translation QLMs (Berger and Lafferty, 1999), and latent variable models (Wei and Croft, 2006) have been proposed in order to take semantic relationships between document and target query into account.
Recently, word embeddings, which are continuous vector representations embedding word semantic information, have been utilized for enhancing the previous expansion techniques (Zhang et al., 2016;Mitra and Craswell, 2017). The word embeddings can be easily acquired in an unsupervised manner from large scale text sets based on embedding modeling, i.e., skip-gram, continuous bag-of-words (CBOW) (Mikolov et al., 2013) or GloVe (Pennington et al., 2014). Zuccon et al. (2015); Ganguly et al. (2015); Zamani and Croft (2016a) used the word embeddings in order to assist translation QLMs. Zamani and Croft (2016b); Kuzi et al. (2016) used the word embeddings in order to perform query expansion. However, previous word embedding-based probabilistic IR methods have no theoretical validity since the word embeddings were heuristically introduced.
In order to perform more natural word embedding based probabilistic IR, our key idea is to directly leverage word embeddings rather than words as random variables for language models. In fact, the word embeddings can capture semantic similarity of words using directional information based on cosine distance (Mnih and Kavukcuglu, 2013). This motivates us to introduce directional probabilistic models based on von Mises-Fisher distributions which are familiar to the cosine distance (Banerjee et al., 2005;Sra, 2016). This paper proposes a hyperspherical QLMs in which random variables are modeled by a mix-ture of von Mises-Fisher distributions. The hyperspherical QLMs can theoretically utilize word embeddings for probabilistic IR without introducing heuristic formulations. Main contributions are summarized as follows.
• This paper formulates hyperspherical QLMs based on both a maximum likelihood estimation and a maximum a posteriori estimation. • This paper reveals that hyperspherical QLMs can be represented as an extended form of categorical QLMs and a theoretical form of translation QLMs. • This paper shows document retrieval evaluation results in which a hyperspherical QLM is compared with conventional QLMs and document distance metrics with word or document embeddings.

Related Work
This paper is closely related to document distance metrics using word or document embeddings. One major distance metric is the cosine distance between two document vectors that are composed by averaging word embeddings (Vulic and Moens, 2015;Brokos et al., 2016) or document embeddings called paragraph vectors (PVs) (Le and Mikolov, 2014). Another highly efficient distance metric is word mover's distance (WMD), which leverages word embeddings (Kusner et al., 2015) In this work, we also examined these distance metrics in a document retrieval evaluation. Generative models of word embeddings have recently been proposed in topic modeling in order to capture the semantic structure of words and documents (Das et al., 2015;Batmanghelich et al., 2016). To the best of our knowledge, this paper is the first work on language modeling that handles word embeddings as random variables.

IR based on QLMs
IR based on probabilistic modeling uses the probability of a document D given a query Q. One of the most famous approaches is QLM-based IR in which documents are ranked by the probabilities that a query can be generated by the document language model (Ponte and Croft, 1998). Given a query Q = {w 1 , · · · , w T }, IR based on QLMs ranks documents as where Θ D denotes a parameter of QLM for D.

Categorical QLMs
Categorical QLMs model P (w|Θ D ) using a categorical distribution. In a maximum likelihood (ML) estimation for the categorical QLM, a generative probability of a word w is defined as where c(w, D) is the word count of w in D, and |D| is the number of all words in D.
In a maximum a posteriori (MAP) estimation, a document collection C in which all of the retrieved documents are included is used for a prior (Zhai and Lafferty, 2001). MAP estimated generative probability of a word w is defined as where c(w, D) is the word count of w in C, and |C| is the number of all words in C. τ is a hyper parameter for adjusting smoothing.

Translation QLMs
Translation QLMs were introduced for expanding categorical QLMs (Berger and Lafferty, 1999). The translation QLMs are usually used together with the categorical QLMs, and enable us to take into account relationships between a word in the query and semantically related words in the document. A generative probability of a word w is defined as where V is the vocabulary. P (v|Θ D ) is the generative probability of a word v, which is also calculated by Eq.
(2) or (3). P (w|v) represents the probability of translating word v into word w. P (w|v) is heuristically calculated as where sim(w, v) is the word similarity between w and v. In order to calculate P (w|v), cosine distances between pre-trained word embeddings were recently utilized (Zuccon et al., 2015;Ganguly et al., 2015). Thus, the word similarity is calculated as where w is the word embedding normalized to a unit length for w.

IR based on Hyperspherical QLMs
This paper proposes a novel probabilistic IR method based on hyperspherical QLM that leverages pre-trained word embeddings as random variables for directional probabilistic models. The pre-trained word embeddings can capture semantic similarity of words on a unit hypersphere, so we deal with normalized word embeddings to unit length. Given a query Q = {w 1 , · · · , w T }, normalized word embeddings {w 1 , · · · , w T } can be acquired using embedding models. IR based on hyperspherical QLMs ranks documents as where p means a probability density and Λ D denotes a parameter of a hyperspherical QLM for D.

Formulation
The hyperspherical QLMs are formulated by a mixture of von Mises-Fisher distributions which are familiar to cosine distances (Banerjee et al., 2005;Sra, 2016). The von Mises-Fisher distribution defines a probability density over points on a unit hypersphere. A probability density based on the mixture of von Mises-Fisher distributions is formulated as where M is the number of mixtures. The parameter Λ corresponds to {α m , µ m }. α m means a mixture weight, and µ m means a directional mean of the m-th von Mises-Fisher distribution. κ is a concentration parameter that is treated as a smoothing parameter. C d (κ) is a normalized parameter that depends on κ and the number of dimensions of word embeddings d. Note that w ⊤ µ m is the cosine distance between w and µ m .

ML and MAP Estimation for Mixture of von Mises-Fisher Distributions
ML estimation for a mixture of von Mises-Fisher distributions given normalized word embeddings D = {w 1 , · · · , w |D| } determines model parameters Λ ML D as The ML estimation is based on the expectation maximization algorithm. ML estimated parameters Λ ML D = {α m }, {μ m } are recursively calculated asα where q(m|w t , Λ) is a load factor of the m-th distribution for the t-th word embedding. MAP estimation for a mixture of von Mises-Fisher distributions given normalized word embeddings D = {w 1 , · · · , w |D| } determines model parameters Λ ML D as Given pre-trained parametersΛ = {ᾱ m ,r m }, where τ is a hyper parameter for adjusting smoothing. For computation of load factors, both the softassignment rule and the hard-assignment rule can be used. Both computations are defined as where q s is the load factor using the softassignment rule and q h is the load factor using the hard-assignment rule.

Training of Hyperspherical QLMs
In order to introduce a mixture of von Mises-Fisher distributions to hyperspherical QLMs, unified assumptions are essential for each document modeling because the hyperspherical QLMs are utilized for IR. Therefore, we introduce the following assumptions.
• The number of mixtures corresponds to vocabulary size.
• The mean direction of each von Mises-Fisher distribution is fixed to normalized word embeddings of each word in the vocabulary.
where v m ∈ V.
• For computation of load factors in document modeling, a hard-assignment rule is used.
To summarize the above, the hyperspherical QLMs can be theoretically estimated as simple forms using ML or MAP estimation. In fact, mixture weights are estimated as ML or MAP estimated values in categorical QLMs. In ML estimation,α m estimated from document D is determined asα Thus, ML estimated generative probability of w in the hyperspherical QLMs is formulated as where v is a normalized word embedding of v.
In MAP estimation,α m estimated from document D is determined aŝ where C is document collection in which all retrieved documents are included. Thus, MAP estimated generative probability of w in the hypersherical QLMs is formulated as

Relationships
The hyperspherical QLMs can be interpreted as an extended form of categorical QLMs. Eq. (23) includes the ML estimated term presented in Eq.
(3). In fact, hyperspherical QLMs can be converted into categorical QLMs by In addition, hyperspherical QLMs can be regarded as a theoretical form of translation QLMs. Eqs. (23) and (25) are similar to Eq. (4). In fact, hyperspherical QLMs are almost the same as translation QLMs by defining a word similarity as While Eq. (6) is heuristically formulated, Eq. (27) is theoretically formulated as a log-linear form based on the directional probabilistic modeling.

Setups
We performed an experiment on a document retrieval task, in which we used 20 news group datasets 1 for evaluation. The datasets were formally split into 11,314 training and 7,531 test articles. The training articles were used for collecting documents and the test articles were used for queries. Label information about news groups was only utilized for deciding whether a retrieved document is relevant to the query in evaluation. These setups are equivalent to the evaluation in Salakhutdinov and Hinton (2009); Larochelle and Lauly (2012). We removed common stop words, and the 5,000 most frequent words in the training articles were used for the vocabulary. In order to utilize word embeddings, data sets in a one billion word language modeling benchmark 2 were prepared. We constructed CBOW and PV with distributed BOW (PV-DBOW). These pretrained embedding models were utilized for IRmethods. The dimension of the word and document embeddings was set to 200.

Results
We assessed the performance of each IR-method with precision-recall curves, mean average precision (mAP), and precision at 10 (P@10). Figure 1 shows the precision-recall curve results and Table  1 shows the mAP and P@10 results. The results are averaged over all possible queries. The results show that CQLM and HQLM clearly outperformed document distance metricbased IR methods with word or document embeddings. This confirms that QLM-based IR is a helpful approach for document retrieval. Although word or document embeddings trained from a lot of text sets can efficiently capture semantic information in continuous space, the document distance metrics using the embeddings were insufficient for document retrieval. In addition, we attained superior performance only introducing HQLM compared to CQLM while single use of TQLM did not perform well. In fact, previous work attained performance improvements by combining TQLM with CQLM. These results confirm that HQLM can effectively utilize word embeddings for document retrieval. Furthermore, we analyzed that relevant documents ranked low in CQLM were moved up by HQLM. This indicates HQLM can robustly calculate generative probabilities of words that were not included in a target query. We also verified that HQLM showed similar results to CQLM when κ was set to a large value (κ = 500). This confirms that Eq. (12), which insists HQLM can be represented as CQLM, is a proper theory.

Conclusions
In this paper, we proposed a word embeddingbased probabilistic IR method based on hyperspherical QLMs that are modeled by a mixture of von Mises-Fisher distributions. We found the hyperspherical QLMs could theoretically utilize word embeddings for IR without introducing heuristic formulations. We found that the hyperspherical QLMs can be represented as an extended form of categorical QLMs and a theoretical form of translation QLMs. Our experiments on a document retrieval task showed hyperspherical QLM outperformed previous QLMs and document distance metrics with word or document embeddings.
In the future, we will examine large scale document retrieval evaluation.