{'fraction_non_alphanumeric': 0.04230436038117239, 'fraction_numerical': 0.009998556165174704, 'mean_word_length': 4.671443193449335, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Latent Semantic Matching: Application to Cross-language Text Categorization without Alignment Information

Unsupervised object matching (UOM) is a promising approach to cross-language natural language processing such as bilingual lexicon acquisition, parallel corpus construction, and cross-language text categorization, because it does not require labor-intensive linguistic resources. However, UOM only finds one-to-one correspondences from data sets with the same number of instances in source and target domains, and this prevents us from applying UOM to real-world cross-language natural language processing tasks. To alleviate these limitations, we proposes latent semantic matching, which embeds objects in both source and target language domains into a shared latent topic space. We demonstrate the effectiveness of our method on cross-language text categorization. The results show that our method outperforms conventional unsupervised object matching methods.


Introduction
Unsupervised object matching is a method for finding one-to-one correspondences between objects across different domains without knowledge about the relation between the domains. Kernelized sorting (Novi et al., 2010) and canonical correlation analysis based methods (Haghighi et al., 2008;Tripathi et al., 2010) are two such examples of unsupervised object matching, which have been shown to be quite useful for cross-language natural language processing (NLP) tasks. One of the most important properties of the unsupervised object matching is that it does not require any linguistic resources which connects between the languages. This distinguishes it from other crosslanguage NLP methods such as machine transla-tion based and projection based approaches (Dumais et al., 1996;Gliozzo and Strapparava, 2005;Platt et al., 2010), which we need bilingual dictionaries or parallel sentences.
When we apply unsupervised object matching methods to cross-language NLP tasks, there are two critical problems. The first is that they only find one-to-one matching. The second is they require the same size of source-and target-data. For example, the correct translation of a word is not always unique. French words 'maison', 'appartment' and 'domicile' can be regarded as translation of an English word 'home'. In addition, English vocabulary size is not equal to that of French.
These discussions motivate us to introduce a shared space in which both source and target domain objects will reside. If we can obtain such a shared space, we can match objects within the space, because we can use standard distance metrics on this space. This will also enable us to use various kinds of non-strict matching. For example, k-nearest objects in the source domain will be retrieved for a query object in the target domain. In this paper, we propose a simple but effective method to find the shared space by assuming that two languages have common latent topics, which we call latent semantic matching. With latent semantic matching, we first find latent topics in two domains independently. Then, the topics in two domains are aligned by kernelized sorting, and objects are embedded in a shared latent topic space. Latent topic representations are successfully used in a wide range of NLP tasks, such as information retrieval and text classification, because they represent intrinsic information of documents (Deerwester et al., 1990). By matching latent topics, we can find relation between source and target domains, and additionally we can handle different numbers of objects in two domains.
We compared latent semantic matching with conventional unsupervised object matching meth-ods on the task of cross-language text categorization, i.e. classifying target side unlabeled documents by label information obtained from source side documents. The results show that, with more source side documents, our method achieved the highest classification accuracy.

Related work
Many cross-language text processing methods have been proposed that require correspondences between source and target languages. For example, (Dumais et al., 1996) proposed cross-lingual latent semantic indexing, and (Platt et al., 2010) employed oriented principle component analysis and canonical correlation analysis (CCA). They concatenate the document pairs (source document and its translation) obtained from a documentlevel parallel corpus. They then apply multivariate analysis to acquire the translingual projection. There are extensions of latent Dirichlet allocation (LDA) (Blei et al., 2003) for cross-language analysis, such as multilingual topic models (Boyd-Graber and Blei, 2009), joint LDA (Jagadeesh and Daume III, 2010) and multilingual LDA (Xiaochuan et al., 2011). They require a bilingual dictionary or document-level parallel corpora.
Unsupervised object matching methods have been proposed recently (Novi et al., 2010;Haghighi et al., 2008;Yamada and Sugiyama, 2011). These methods are promising in terms of language portability because they do not require external language resources. (Novi et al., 2010) proposed kernelized sorting (KS); it finds one-toone correspondences between objects in different domains by permuting a set to maximize the dependence between two sets. Here, the Hilbert-Schmidt independence criterion is used for measuring dependence. (Djuric et al., 2012) proposed convex kernelized sorting as an extension of KS. (Yamada and Sugiyama, 2011) proposed leastsquares object matching which maximizes the squared-loss mutual information between matched pairs. (Haghighi et al., 2008) proposed another framework, matching CCA (MCCA), based on a probabilistic interpretation of CCA (Bach and Jordan, 2005). MCCA simultaneously finds latent variables that represent correspondences and latent features so that the latent features of corresponding examples exhibit the maximum correlation. However, these unsupervised object matching methods have limitations. They require that the source and target domains have the same data size, and they find one-to-one correspondences. There are critical weaknesses of these methods when we attempt to apply them to real world cross-language NLP applications.

Latent Semantic Matching
We propose latent semantic matching to find a shared latent space by assuming that two languages have common latent topics. Our method consists of following four steps: (1) for both source and target domains, we map the documents to a K-dimensional latent topic space independently, (2) we find the one-to-one correspondences between topics across source and target domains by unsupervised object matching, (3) we permute topics of the target side according to the correspondences, while fixing the topics of the source side, and (4) finally, we map documents in the source and target domains to a shared latent space by using permuted and fixed topics.

Topic Extraction as Dimension Reduction
Suppose that we have N documents in the source domain. s n =(s ni ) I i=1 is the nth document represented as a multi-dimensional column vector in the domain, i.e. each document is represented as a bag-of-words vector. Here, each element of the vectors indicates the TF·IDF score of the corresponding word in the document. I is the size of the feature set, i.e., the vocabulary size in the source domain. Also, we have M documents in the target domain. t m =(t mj ) J j=1 is the mth document represented as a multi-dimensional vector. J is the vocabulary size in the target domain. Thus, the data set in the source domain is represented by an I × N matrix, S=(s 1 , · · · , s N ), the data set in the target is represented by a J × M matrix, T =(t 1 , · · · , t M ).
We factorize these matrices using nonnegative matrix factorization (Lee and Seung, 2000) to find topics as follows:  According to the factorization of the original matrices, we can map the documents in the source and target domain to latent topic space with dimentionaly K, independently.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent space, the basis of the space have to be aligned each other. That is, topic proportion extracted from the source language must be aligned that from the target language. This is reasonable consideration because we can assume the same latent concept for both language. For example, a topic proportion obtained from English documents can be aligned a topic proportion obtained from French documents. For all k and k , k-th column vector in WS are aligned k -th column vector in WT . However, we can not measure similarity between the topic proportions because we do not have any language resources such as dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topic proportions. In this paper, we employ Kernelized Sorting (KS) [Novi et al., 2010]. Of cource, we can replace KS to another unsupervised object matching sush as MCCA [Haghighi et al., 2008], LSOM [Yamada and Sugiyama, 2011].
KS finds the best one-to-one matching by followings: s.t. π1K = 1K and π T 1K = 1K . (3) π is K × K matrix which represents one-to-one correspondence between topic proportion, i.e., πij = 1 indicates i-th topic proportion in the source language corresponds to j-th β is cost parameter between first, second argument and third argument.
The optimal parameters (WS, WT , HS, HT ) are obtained by minimizing the objective function. To mimimize the objective, gradient descend can be used. but However that is not convex function, we only obtained local optimal. Thefore, we employed above two step procedure??????
This objective function is not convex. That means we can only obtain local optimal parameters. By minimizing equation (6), we can obtain a set of parameter (WS, WT , HS, HT ) for unsupervised mapping. we could be employed gradient based algorithm but, as the first step, we employ former two step optimization procedure.

Cross-lingual Text Categorization via
Unsupervised Mapping m-th document in the target domain (tm) is mapped to the source domain as follows, s(tm) = HT :,mWS .
Here, HT :,m denotes the m-th column vector of HT , s(tm) is I dimentional vector. When each document in the source domain has a class label yn, we can train a classifier on the training data set {sn, yn} N n=1. Therefore, the class label of the mapped document in the target domain s(tm) is assigned by the classifier. In the later experiments, we employ k(= 10)-NN as a classifier.
2000] to factorize the original matrices.
According to the factorization of the original matrices, we can map the documents in the source and target domain to latent topic space with dimentionaly K, independently.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent space, the basis of the space have to be aligned each other. That is, topic proportion extracted from the source language must be aligned that from the target language. This is reasonable consideration because we can assume the same latent concept for both language. For example, a topic proportion obtained from English documents can be aligned a topic proportion obtained from French documents. For all k and k , k-th column vector in WS are aligned k -th column vector in WT . However, we can not measure similarity between the topic proportions because we do not have any language resources such as dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topic proportions. In this paper, we employ Kernelized Sorting (KS) [Novi et al., 2010]. Of cource, we can replace KS to another unsupervised object matching sush as MCCA [Haghighi et al., 2008], LSOM [Yamada and Sugiyama, 2011].
KS finds the best one-to-one matching by followings: s.t. π1K = 1K and π T 1K = 1K . (3) π is K × K matrix which represents one-to-one correspondence between topic proportion, i.e., πij = 1 indicates i-th topic proportion in the source language corresponds to j-th β is cost parameter between first, second argument and third argument.
The optimal parameters (WS, WT , HS, HT ) are obtained by minimizing the objective function. To mimimize the objective, gradient descend can be used. but However that is not convex function, we only obtained local optimal. Thefore, we employed above two step procedure??????
This objective function is not convex. That means we can only obtain local optimal parameters. By minimizing equation (6), we can obtain a set of parameter (WS, WT , HS, HT ) for unsupervised mapping. we could be employed gradient based algorithm but, as the first step, we employ former two step optimization procedure.

Cross-lingual Text Categorization via
Unsupervised Mapping m-th document in the target domain (tm) is mapped to the source domain as follows, s(tm) = HT :,mWS .
Here, HT :,m denotes the m-th column vector of HT , s(tm) is I dimentional vector. When each document in the source domain has a class label yn, we can train a classifier on the training data set {sn, yn} N n=1. Therefore, the class label of the mapped document in the target domain s(tm) is assigned by the classifier. In the later experiments, we employ k(= 10)-NN as a classifier.
2000] to factorize the original matrices.
According to the factorization of the original matrices, we can map the documents in the source and target domain to latent topic space with dimentionaly K, independently.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent space, the basis of the space have to be aligned each other. That is, topic proportion extracted from the source language must be aligned that from the target language. This is reasonable consideration because we can assume the same latent concept for both language. For example, a topic proportion obtained from English documents can be aligned a topic proportion obtained from French documents. For all k and k , k-th column vector in WS are aligned k -th column vector in WT . However, we can not measure similarity between the topic proportions because we do not have any language resources such as dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topic proportions. In this paper, we employ Kernelized Sorting (KS) [Novi et al., 2010]. Of cource, we can replace KS to another unsupervised object matching sush as MCCA [Haghighi et al., 2008], LSOM [Yamada and Sugiyama, 2011].
KS finds the best one-to-one matching by followings: s.t. π1K = 1K and π T 1K = 1K . (3) π is K × K matrix which represents one-to-one correspondence between topic proportion, i.e., πij = 1 indicates i-th topic proportion in the source language corresponds to j-th β is cost parameter between first, second argument and third argument.
The optimal parameters (WS, WT , HS, HT ) are obtained by minimizing the objective function. To mimimize the objective, gradient descend can be used. but However that is not convex function, we only obtained local optimal. Thefore, we employed above two step procedure??????
This objective function is not convex. That means we can only obtain local optimal parameters. By minimizing equation (6), we can obtain a set of parameter (WS, WT , HS, HT ) for unsupervised mapping. we could be employed gradient based algorithm but, as the first step, we employ former two step optimization procedure.

Cross-lingual Text Categorization via
Unsupervised Mapping m-th document in the target domain (tm) is mapped to the source domain as follows, s(tm) = HT :,mWS .
Here, HT :,m denotes the m-th column vector of HT , s(tm) is I dimentional vector. When each document in the source domain has a class label yn, we can train a classifier on the training data set {sn, yn} N n=1. Therefore, the class label of the mapped document in the target domain s(tm) is assigned by the classifier. In the later experiments, we employ k(= 10)-NN as a classifier.
2000] to factorize the original matrices.
According to the factorization of the original matrices, we can map the documents in the source and target domain to latent topic space with dimentionaly K, independently.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent space, the basis of the space have to be aligned each other. That is, topic proportion extracted from the source language must be aligned that from the target language. This is reasonable consideration because we can assume the same latent concept for both language. For example, a topic proportion obtained from English documents can be aligned a topic proportion obtained from French documents. For all k and k , k-th column vector in WS are aligned k -th column vector in WT . However, we can not measure similarity between the topic proportions because we do not have any language resources such as dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topic proportions. In this paper, we employ Kernelized Sorting (KS) [Novi et al., 2010]. Of cource, we can replace KS to another unsupervised object matching sush as MCCA [Haghighi et al., 2008], LSOM [Yamada and Sugiyama, 2011].
(3) π is K × K matrix which represents one-to-one correspondence between topic proportion, i.e., πij = 1 indicates i-th topic proportion in the source language corresponds to j-th β is cost parameter between first, second argument and third argument.
The optimal parameters (WS, WT , HS, HT ) are obtained by minimizing the objective function. To mimimize the objective, gradient descend can be used. but However that is not convex function, we only obtained local optimal. Thefore, we employed above two step procedure??????
This objective function is not convex. That means we can only obtain local optimal parameters. By minimizing equation (6), we can obtain a set of parameter (WS, WT , HS, HT ) for unsupervised mapping. we could be employed gradient based algorithm but, as the first step, we employ former two step optimization procedure.

Cross-lingual Text Categorization via
Unsupervised Mapping m-th document in the target domain (tm) is mapped to the source domain as follows, s(tm) = HT :,mWS .
Here, HT :,m denotes the m-th column vector of HT , s(tm) is I dimentional vector. When each document in the source domain has a class label yn, we can train a classifier on the training data set {sn, yn} N n=1. Therefore, the class label of the mapped document in the target domain s(tm) is assigned by the classifier. In the later experiments, we employ k(= 10)-NN as a classifier.
2000] to factorize the original matrices.
According to the factorization of the original matrices, we can map the documents in the source and target domain to latent topic space with dimentionaly K, independently.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent space, the basis of the space have to be aligned each other. That is, topic proportion extracted from the source language must be aligned that from the target language. This is reasonable consideration because we can assume the same latent concept for both language. For example, a topic proportion obtained from English documents can be aligned a topic proportion obtained from French documents. For all k and k , k-th column vector in WS are aligned k -th column vector in WT . However, we can not measure similarity between the topic proportions because we do not have any language resources such as dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topic proportions. In this paper, we employ Kernelized Sorting (KS) [Novi et al., 2010]. Of cource, we can replace KS to another unsupervised object matching sush as MCCA [Haghighi et al., 2008], LSOM [Yamada and Sugiyama, 2011].
KS finds the best one-to-one matching by followings: s.t. π1K = 1K and π T 1K = 1K . (3) π is K × K matrix which represents one-to-one correspondence between topic proportion, i.e., πij = 1 indicates i-th topic proportion in the source language corresponds to j-th β is cost parameter between first, second argument and third argument.
The optimal parameters (WS, WT , HS, HT ) are obtained by minimizing the objective function. To mimimize the objective, gradient descend can be used. but However that is not convex function, we only obtained local optimal. Thefore, we employed above two step procedure??????
This objective function is not convex. That means we can only obtain local optimal parameters. By minimizing equation (6), we can obtain a set of parameter (WS, WT , HS, HT ) for unsupervised mapping. we could be employed gradient based algorithm but, as the first step, we employ former two step optimization procedure.

Cross-lingual Text Categorization via
Unsupervised Mapping m-th document in the target domain (tm) is mapped to the source domain as follows, s(tm) = HT :,mWS .
Here, HT :,m denotes the m-th column vector of HT , s(tm) is I dimentional vector. When each document in the source domain has a class label yn, we can train a classifier on the training data set {sn, yn} N n=1. Therefore, the class label of the mapped document in the target domain s(tm) is assigned by the classifier. In the later experiments, we employ k(= 10)-NN as a classifier. vector denotes an embedding of a document in the K-dimensional latent space. Similarly, W T is an I × K matrix that represents a set of topics in the target domain, and H T is a K × M matrix that denotes a set of latent semantic representations of target documents. K is less than I and J. By factorizing the original matrices, we can independently map the documents in the source and target domains to the latent topic spaces whose dimensionality is K.

Finding Optimal Topic Alignments by Unsupervised Object Matching
To connect the different latent spaces, topics extracted from the source language must be aligned to one from the target language. This is reasonable because we can assume that both languages share the same latent concept. However, we cannot quantify the similarity between the topics because we do not have any external language resources such as a dictionary. Therefore, we utilize unsupervised object matching method to find one-to-one correspondences between topics. In this paper, we employ kernelized sorting (KS) (Novi et al., 2010). KS finds the best one-to-one matching as follows: Here, π is a K × K matrix that represents the oneto-one correspondence between topics, i.e. π ij =1 indicates that the ith topic in the source language corresponds to the jth one of the target language.  Table 1: Average accuracy over all language pairs Π K indicates the set of all possible matrices storing one-to-one correspondences. G denotes the K × K kernel matrix obtained from topic proportion, G ij =K(W ⊤ i,: , W :,j ), and G is the centered matrix of G. K(, ) is a kernel function. 1 K is a K-dimensional column vector of all ones. π * is obtained by iterative procedure.
According to π * , we obtain permuted matrices, W T =W T π * and H T =π * ⊤ H T , and the product of permuted matrices is the same with that of unpermuted matrices as follows:

Cross-language Text Categorization via Latent Semantic Matching
Cross-language text categorization is the task of exploiting labeled documents in the source language (e.g. English) to classify documents in the target language (e.g. French). Suppose we have training data set {s n , y n } N n=1 in the source language domain. y n ∈ Y is the class label for the nth document. We can train a classifier in the K-dimensional latent space with data set {H ⊤ S:,n , y n } N n=1 . H ⊤ S:,n is the projected vector of s n . Also, the mth document in the target language domain t m is projected into the latent space as H ⊤ T :,m . Here, the documents in both domains are projected into the same size latent space and the basis vectors of the spaces are aligned. Therefore, we can classify a document in the target domain t m by a classifier trained with {H ⊤ S:,n , y n } N n=1 . We compared our method, latent semantic matching (LSM), with three unsupervised object matching methods: Kernelized Sorting (KS), Convex Kernelized Sorting (CKS), Least-Squares Object Matching (LSOM). We set the number of the latent topics K to 100 and employed the k-nearest neighbor method (k=10) as the classifier. For, KS, CKS and LSOM, we find the oneto-one correspondence between documents in the source language and documents in the target language. Then, we assign class labels of the target documents according to the correspondence.
In order to build a corpus with various language pairs for evaluation, we crawled product reviews from Amazon U.S., German, France and Japan with five categories: 'Books', 'Electronics', 'Music', 'Kitchen', 'Watch'. The corpus is neither sentence level parallel nor comparable. For each category, we randomly select 60 documents as the test data (M =300) for all methods and 60 documents as the training data (N =300) for KS, CKS, LSOM and LSM(300). We also compared latent semantic matching with 120 training documents for each category (N =600), and called this method LSM(600). Note that since KS, CKS and LSOM require that the data sizes are the same for source and target domains, they cannot use training data more than test data. To avoid local optimum solutions of NMF, we executed our methods with 100 different initialization values and chose the solution that achieved the best objective func-tion of KS. Table 1 shows average accuracies with standard division over all language pairs. From the table, classification accuracy of all methods significantly outperformed random classifier (accuracy=0.2). The results showed the effectiveness of both unsupervised object matching and latent semantic matching. When comparing LSM(300) with KS, CKS and LSOM, LSM(300) obtained better results than these unsupervised object matching methods. The result supports the effectiveness of the latent topic matching. Moreover, LSM(600) achieved the highest accuracy. There are large differences between LSM(600) and the others. This result implies not only the effectiveness of the latent topic matching but also increasing the number of source side documents (labeled training data) contributes to improving classification accuracy. This is natural in terms of supervised learning but only our method can deal with source side documents that are larger in number. Table 2 shows examples of latent topics in English and German extracted and aligned by LSM(600). We can see that some author names, words related to camera, and cooking equipment appear in 'Books', 'Electronics' and 'Kitchen' topics, respectively. Similarity, there are some artists' names in 'Music' and watch brands in 'Watch'.

Conclusion
As an extension of unsupervised object matching, this paper proposed latent semantic matching that considers the shared latent space between two language domains. To generate such a space, topics of the target space are permuted by exploiting unsupervised object matching. We can measure distances between objects by standard metrics, which enable us retrieving k-nearest objects in the source domain for a query object in the target domain. This is a significant advantage over conventional unsupervised object matching methods. We used Amazon review corpus to demonstrate the effectiveness of our method on crosslanguage text categorization. The results showed that our method outperformed conventional object matching methods with the same number of training samples. Moreover, our method achieved even higher performance by utilizing more documents in the source domain.