{'fraction_non_alphanumeric': 0.036946184617707, 'fraction_numerical': 0.014604875839686013, 'mean_word_length': 4.602325581395349, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Context Reinforced Neural Topic Modeling over Short Texts

As one of the prevalent topic mining tools, neural topic modeling has attracted a lot of interests for the advantages of high efficiency in training and strong generalisation abilities. However, due to the lack of context in each short text, the existing neural topic models may suffer from feature sparsity on such documents. To alleviate this issue, we propose a Context Reinforced Neural Topic Model (CRNTM), whose characteristics can be summarized as follows. Firstly, by assuming that each short text covers only a few salient topics, CRNTM infers the topic for each word in a narrow range. Secondly, our model exploits pre-trained word embeddings by treating topics as multivariate Gaussian distributions or Gaussian mixture distributions in the embedding space. Extensive experiments on two benchmark datasets validate the effectiveness of the proposed model on both topic discovery and text classification.


Introduction
Mining topics from texts is significant for various applications of natural language processing, e.g., text classification, sentiment analysis, and recommender systems. As one of the most popular approaches for discovering latent topics, topic modeling (Blei et al., 2003;Yin and Wang, 2014) is capable of producing interpretable results. Generally, the dominant methods for parameter estimation in topic models are variational inference (Blei et al., 2003) and Gibbs sampling (Griffiths and Steyvers, 2004), both of which, however, require complex rederivation when there is any minor changes to the model structure. Moreover, with the growth of data scale, the generative process is getting tricky and expensive, which leads to mathematically arduous derivation and high computational cost in training. These limitations make it difficult to extend the models to new variations flexibly. * The corresponding author.
With the development of deep learning, variational auto-encoder (VAE) (Kingma and Welling, 2014) has provided another promising solution for topic modeling. Benefiting from the flexibility of neural networks, the VAE framework is competent to learn complicated non-linear distributions and is convenient to be applied to various tasks. Furthermore, by using the back-propagation for optimization, VAE is highly efficient in training when compared with the models based on variational inference or Gibbs sampling. Considering the above advantages, several models built on VAE have been proposed, such as neural variational document model (NVDM) (Miao et al., 2016), neural variation latent Dirichlet allocation (NVLDA) (Srivastava and Sutton, 2017), Gaussian softmax model (GSM) (Miao et al., 2017), Dirichlet variational auto-encoder (DVAE) (Burkhardt and Kramer, 2019), and neural variational correlated topic modeling (NVCTM) (Liu et al., 2019). Although the VAE-based models reduce the computational cost impressively, they still suffer from the feature sparsity problem in short texts. In this case, the number of word occurrences in each text is relatively small, while the vocabulary corresponding to the corpus is large and the range of topics is broad.
To alleviate the above issue, many Bayesian approaches specific to short texts have been proposed (Yan et al., 2013;Lin et al., 2014;Li et al., 2016). Nonetheless, the above models all resort to Gibbs sampling or variational inference and hence incur the problems as mentioned before. In recent years, models built on VAE are also introduced for short texts, such as Graph-based inference network for the biterm topic model (GraphBTM) (Zhu et al., 2018) and neural sparsemax topic model (NSMTM) (Lin et al., 2019). However, learning context information is still challenging in these models due to significant word non-overlap in short texts. Relatedness information between word pairs may not be fully captured owing to the lack of word-overlap between such short messages.
In this paper, we propose a VAE-based topic model for short texts, where the context information for each text is effectively enhanced. Firstly, as can be observed, a short text generally covers only a subset of topics due to the limited text length. Therefore, we propose to filter irrelevant topics by setting a topic controller for each topic, encouraging each short text to focus on some salient topics. Through this way, the topic inference range is narrowed down and thus the topic sparsity can be achieved indirectly. Secondly, we incorporate pre-trained word embeddings into our model to explicitly enrich the context information. Specifically, we model each topic by a multivariate Gaussian distribution or a Gaussian mixture distribution in the embedding space, through which the relatedness of synonymous word pairs can be effectively inferred regardless of word non-overlap in short texts. In this way, our model can discover more interpretable topics than other topic models. We name the proposed model as Context Reinforced Neural Topic Model (CRNTM) and conclude the main contributions of our work as follows: • We assume that each short text only focuses on a few salient topics. By setting a topic controller for each topic to filter irrelevant topics, CRNTM narrows down the topic inference space and achieves topic sparsity indirectly.
• Pre-trained word embeddings are incorporated to explicitly enrich the limited context information for each short message. By treating topic distributions over words as multivariate Gaussian distributions or Gaussian mixture distributions in the embedding space, CRNTM can produce more interpretable topics.
The rest of this paper is organized as follows. We discuss relevant research work in Section 2, and detail our proposed model in Section 3. Experimental settings and results are presented in Section 4. Finally, we draw the conclusion in Section 5.

Neural Topic Modeling
With the development of deep learning, models built on neural networks have been proposed to discover latent topics, and most of them are based on VAE. In this vein, NVDM (Miao et al., 2016) is a neural variational framework for generative modeling on texts. It consists of an inference network and a multinomial softmax generative module. The inference network is used to estimate continuous hidden variables, which can represent the semantic content of documents, while the generative module aims to reconstruct the documents from the latent topic distributions. GSM (Miao et al., 2017) constructs the topic distributiona explicitly with a softmax function applied to the projection of the Gaussian random vector. ProdLDA (Srivastava and Sutton, 2017) replaces the mixture model in latent Dirichlet allocation (LDA) with a product of experts for better topic modeling. NVLDA (Srivastava and Sutton, 2017) approximates the Dirichlet distribution by using Laplace approximation. DVAE (Burkhardt and Kramer, 2019) decouples the properties of sparsity and smoothness by rewriting the Dirichlet parameter vector into a product of a sparse binary vector and a smoothness vector. NVCTM (Liu et al., 2019) enhances the capability of capturing the correlations among topics by reshaping topic distributions.

Short Text Topic Discovery
Topic models (Blei et al., 2003) provide a valuable solution for implicit semantic mining and understanding over documents. However, the feature sparsity problem arises for topic models when applied to short texts , because such corpora are lack of word co-occurrences at the document level.
To overcome this limitation, the external documents were first introduced to enrich the contextual information in short texts (Sahami and Heilman, 2006;Jin et al., 2011;Phan et al., 2008). Unfortunately, it requires the external documents to be semantically close to the original corpus. Some approaches tackle the task by aggregating short texts into lengthy pseudo-documents and then applying a well established topic model. For this category of methods, short texts can be aggregated by utilizing the side information, e.g., user characteristics tags (Feng et al., 2020), user ID (Zhao et al., 2011, and timestamp (Diao et al., 2012). Another alternative methods directly modify the prior of Bayesian models to enrich word co-occurrences, so as to remedy the feature sparsity problem. For instance, the Biterm Topic Model (BTM) , which models the global word co-occurrences at the corpus level, could lengthen short texts by con-verting documents into biterm sets.
While the above methods are developed based on Bayesian models, some neural network based approaches have been introduced for short texts. Zhu et al. (2018) proposed a graph-based inference network named GraphBTM for accelerating the above BTM. This model sampled a fixed number of texts as a training instance to overcome the feature sparsity issue. Lin et al. (2019) proposed a neural model which is called NSMTM by providing sparse posterior distributions over topics based on the Gaussian sparsemax construction. Gupta et al. (2019) designed a neural autoregressive topic model named iDocNADE in a language modeling fashion. They also incorporated word embeddings as fixed prior in the model to introduce complementary information. However, the above approach does not model topic distributions explicitly.

Model Description
In this section, we describe our context reinforced neural topic model (CRNTM) in details. The overall architecture is illustrated in Figure 1, which consists of two major modules: an inference network for learning latent topics, and a Gaussian decoder for reconstructing documents.

Problem Definition
Given a corpus with D short texts, we denote the corresponding vocabulary as W = {w 1 , w 2 , ..., w V }, with V being the vocabulary size. Following (Miao et al., 2016), each document is processed into a bag-of-words (BOW) representation, i.e., , where x d,i denotes the number of times for word w i appearing in document d.
In the inference network, we use θ d ∈ R K to denote the topic distribution of document d and use z k ∈ {z 1 , z 2 , ..., z K } to denote the topic as-signment for an observed word, where K denotes the number of topics inherent in the given corpus. Specifically, θ d is drawn from the Gaussian distribution N (µ d , Σ d ), where both µ d and Σ d are prior parameters. Furthermore, we set a topic controller λ d,k ∈ [0, 1] for each topic z k in document d: the topic will be kept when λ d,k = 1, or it will be filtered out when λ d, For the Gaussian decoder, we denote the word embedding matrix corresponding to the vocabulary as W E ∈ R V ×r , where r indicates the dimension of word embeddings. Moreover, the embedding of word w i is represented as W E i . We use a matrix T W ∈ R K×V to denote the probabilities of words conditioned to topics, in which, T W (k,i) represents the conditional probability of word w i over topic z k . In this study, T W (k,i) is drawn from a multivariate Gaussian distribution N (µ k , Σ k ) or a Gaussian mixture distribution, where µ k ∈ R r and Σ k ∈ R r×r are learnable parameters.

Inference Network
The first component of CRNTM is the inference network, which is applied to infer the topic distributions for the input documents. The structure of our inference network is illustrated in Figure 2. Following the framework of VAE, CRNTM infers the parameters µ d and Σ d via deep neural networks that are elaborately designed for the observed data. Being fed with the input document x d , the inference network first outputs an encoded vector π d . Then, π d is linearly transformed to obtain µ d and Σ d , which are used to parameterize the Gaussian prior N (µ d , Σ d ). The above process is described by where MLP 1 is a multilayer perceptron, l 1 (·) and l 2 (·) are linear transformations. Note that the diagonal elements σ 2 d of covariance matrix Σ d are non-negative. The output of l 2 (·) is regarded as the logarithmic form log σ d , which is a real number.
A Gaussian random vector h d is passed through a softmax function to parameterize the multinomial document-topic distribution θ d . The process is defined as with reparameterization, allowing the parameters to be optimized by back-propagation. Due to the limited text length, a short document only contains a few of words, resulting in the feature sparsity problem during the inference process. However, it can be observed that a short text generally focuses on a subset of topics. This inspires us to alleviate the above issue by narrowing down the scope for topic inference. Instead of letting the topic mixtures navigate freely in the simplex, CRNTM allows short texts to cover a narrow range of topics. This is achieved by setting a topic controller λ d,k ∈ [0, 1] for each topic. Topic z k is focused when λ d,k = 1, and it will be filtered out when λ d,k = 0. The topic controllers are drawn from the Beta distribution, i.e., λ d ∼ Beta(α d , β d ), as described in Section 3.1, which can guarantee that each component λ d,k ∈ λ d is in the range of [0, 1]. The parameters α d and β d are inferred as follows: where MLP 2 is a multilayer perceptron, l 3 (·) and l 4 (·) are linear transformations. The output of l 3 (·) and l 4 (·) are treated as the logarithmic form log α d and log β d , since both α d and β d are non-negative. The Beta sampling can not be differentiated directly, making it intractable to update model parameters through back-propagation. Therefore, we use the re-parameterization technique to obtain λ d by following (Naesseth et al., 2017). The sampling operation of Beta(α d , β d ) can be decoupled into Gamma(α d , 1) and Gamma(β d , 1), which is formulated by λ d = λ d,1 λ d,1 +λ d,2 , where λ d,1 ∼ Gamma(α d , 1) and λ d,2 ∼ Gamma(β d , 1). For the Gamma distribution Gamma(α, 1) with α > 1, the re-parameterization can be accomplished by the reject sampling method: where d ∼ N (0, I 2 ). On the other hand, the shape augmentation method is applied to convert α ≤ 1 to α > 1 to increase the accept rate of each rejection sampler, which is formulated by , where ρ is drawn from a uniform distribution, i.e., ρ ∼ U [0, 1], andλ d,1 ∼ Gamma(α + 1, 1) can be obtained according to Equation (4).
During the inference process, CRNTM determines whether topic z k is kept according to λ d,k . By filtering out some certain topics, the short texts are allowed to focus on a few specific topics, and thus the feature sparsity problem can be alleviated. Finally, the topic distribution of document x d is obtained by θ d = θ d * λ d .

Gaussian Decoder
Context information is important for topic mining (Gupta et al., 2019). Words that appear together frequently are more likely to belong to the same topic, which implies that closer words in the embedding space are more likely to reflect the same topic. In Bayesian models, word embeddings that are trained on a large corpus have shown to effectively bring auxiliary context information for short texts (Li et al., 2016). Considering this advantage, we propose to introduce word embeddings into the decoder named Gaussian decoder. To our best knowledge, this is the first work of incorporating pre-trained word embeddings into the decoder of VAE to enhance the ability of capturing context information. The basic structure of our Gaussian decoder is shown in Figure 3(a).
The Gaussian decoder is applied to decode the topic distribution θ d , based of which a new document can be reconstructed. Concretely, the decoder employs the multivariate Gaussian distribution N (µ k , Σ k ) to model the k-th topic in the embedding space. Since the elements of the diagonal matrix Σ k are non-negative, a transformation similar to the one applied to Σ d is used here, as follows: Σ k = diag(σ 2 k ). By incorporating pre-trained word embeddings, the probability of word w i conditioned on topic z k , i.e., T W (k,i) , can be formulated by T W (k,i) = exp(g(W E i )) (2π) r/2 |Σ k | 1/2 , where r is the word embedding dimension, and g( It is worth noting that the parameters µ k and Σ k can be regarded as the topic centroid and topic concentration in the embedding space. According to the properties of Gaussian distribution, words that are closer to the topic centroid have higher probabilities. Meanwhile, words in each topic are related to the context information implied by word embeddings. Therefore, CRNTM can enrich context information via pre-trained word embeddings to address the feature sparsity problem. Finally, we estimate the conditional probability p The above method can be easily adjusted by assuming that T W (k,i) obeys a Gaussian mixture distribution, as shown in Figure 3

Optimization Objective
The optimization objective of CRNTM is L = log p(D), where log p(D) is the likelihood of observed samples. According to the assumption, there is log p(D) = d log p(d). Since the true distributions of documents are unknown, variational inference is used here to convert the optimization to its evidence lower bound (ELBO), that is, log p(d) ≥ L(d). According to the variational inference method, L(d) is derived as follows: where For θ d , we assume that the true prior p(θ d ) is a normal Gaussian distribution N (0, I) by following (Kingma and Welling, 2014; Miao et al., 2016;Liu et al., 2019). Therefore, . Similarly, we take Beta(α , β ) as the true prior of p(λ d ), and the KL divergence term Γ(α+β) , Γ(·) is the Gamma function, and ψ(·) is the Digamma function. In our model, the topic controller acts as a switch to filter out irrelevant topics and keep related topics with higher probabilities. Since α and β determine the shape of Beta distribution, we set both α and β to 0.5, so that the probabilities are sharp in values of 0 and 1.

Experiments
In this section, we first introduce the experimental setting, and then evaluate the effectiveness of our model by a series of experiments.

Datasets
To compare the model performance on both topic mining and text classification, we employ 20News- Groups 1 and Snippets 2 with document labels as our datasets. 20NewsGroups is a collection of short news messages, which contains 11, 314 training and 7, 531 testing samples. These short texts are grouped into 20 different categories. Snippets is collected from the results of web search transaction over 8 domain labels. The officially divided 10,060 and 2,280 search transaction documents are used for training and testing, respectively. For data preprocessing, we remove stopwords and take the most frequent 2, 000 words and 5, 000 words as vocabularies. The statistics of the processed corpora are shown in Table 1, where AvgD and L denote the averaged number of words for each document and the number of categories, respectively.

Baseline Methods
We use the following mainstream VAE based methods as baselines for evaluation: NVDM (Miao et al., 2016), NVLDA & ProdLDA (Srivastava and Sutton, 2017), GSM (Miao et al., 2017), TMN (Zeng et al., 2018), NVCTM (Liu et al., 2019), and DVAE (Burkhardt and Kramer, 2019). Among these methods, NVDM is one of the first neural document models, NVLDA, ProdLDA, and GSM are classical neural topic models. TMN consists of a neural topic model and a topic memory mechanism, which are trained in an end-to-end learning manner. NVCTM exploits the Centralized Transformation Flow (CTF) to capture the topic correlations by reshaping topic distributions. DVAE achieves a competitive topic coherence and a high log-likelihood by decoupling the properties of sparsity and smoothness in VAE-based topic models for short texts. Note that iDocNADE (Gupta et al., 2019) and NSMTM (Lin et al., 2019) are not adopted for comparison, because the former does not model topic distributions explicitly while the training process of the latter is too sensitive to continue based on our implementation. Besides, since the ELBO is typically used and necessary to evaluate the performance of VAE based methods (Miao et al., 2016(Miao et al., , 2017, we do not use Bayesian models such as (Yan et al., 2013;Lin et al., 2014;Li et al., 2016) as baselines for fair comparison. Finally, GraphBTM (Zhu et al., 2018) which only models a mini-corpus is unsuitable to be evaluated in this study.

Experimental Settings
In our experiments, the publicly available codes of NVDM 3 , NVLDA & ProdLDA 4 , TMN 5 , and DVAE 6 are directly used. The baselines of GSM and NVCTM are implemented by us based on the code of NVDM, where the length of CTF in NVCTM is set to 10 according to the preliminary experiments. For our model, we use the widely adopted pre-trained word embedding from Glove (Pennington et al., 2014), and the embedding size is 300. All the models are trained alternatively by Adam optimizer with a learning rate of 1e −5 and a batch size of 64.
In the task of topic discovery, perplexity = exp{− 1 } is used to evaluate the generalization performance of models on the testing set, where D is the number of documents, N d is the number of words in document d and p(w d,i ) is the log-likelihood of model on word w i in document d. To evaluate the quality of discovered topics, we also use the normalized pointwise mutual information (NPMI) (Lau et al., 2014) as the metric. The averaged values of NPMI on the top 5, 10, and 15 words for all topics is computed as the final results. For the task of text classification, we use the topic vector of each document generated by convergent models as the input of a classifier. MLPClassifier from scikit-learn 7 is chosen as the classifier in this study and accuracy is used as the metric. For each task, the topic numbers are set to 25, 50, and 100. We denote our models with Gaussian distribution and Gaussian mixture distribution in the decoder as CRNTM GD and CRNTM GMD, respectively. Unless explicitly specified, the number of Gaussian components is set to 25 for CRNTM GMD. The source code, detailed parameter settings, and complementary results of our models can be found at Github 8 . Table 2 presents the test document perplexities of all models, from which we can observe that our CRNTM GD and CRNTM GMD achieve the best results in most cases. Specifically, TMN performs the best on Snippets. The reason may be that TMN is basically a supervised model for text classification and that the supervision from labels can help mining topics on a corpus consisting of less formal texts (i.e., Snippets). We also report the results of topic coherence in Table 3. It can be observed that both of our models are significantly better than all the baselines, which shows that they are able to discover more meaningful and interpretable topics. The performance comparisons for text classification are shown in Table 4. We can find that CRNTM GD and CRNTM GMD obtain competitive performances when compared with the benchmark methods, which validates the effectiveness of our models on generating representative vectors for short text classification.

Evaluation on Gaussian Decoder via Topic Visualization
To investigate the quality of topics discovered by our models, we report top 15 words of 4 representative topics and visualize these topics by their embedding vectors using 20NewsGroups. Particularly, we extract µ k of Gaussian distributions as the topic centroid and utilize t-SNE (van der Maaten, 2009) for visualization. Topic visualization of the results in CRNTM GD is depicted in Figure 4(a). The points with different colors indicate different topics, and the centroid of topic k is denoted as T k. For the convenience of comparison, we manually annotate each topic by referring to the ground truth category. Accordingly, T 1, T 2, T 3, and T 4 in CRNTM GD are annotated as "soc.religion.christian", "talk.politics", "comp.sys.ibm.pc.hardware", and "comp.graphics", respectively. We can see that all top words of the same topics are close to each other and to the corresponding topic centroids in the continuous vector space. This validates that our Gaussian decoder can effectively capture the context information via word embeddings in mining topics. We also present 4 topics generated by CRNTM GMD whose semantics are similar to those in CRNTM GD to verify the effectiveness of Gaussian mixture distributions. Topic visualization of the results in CRNTM GMD is shown in Figure 4(b). For clarity, the coefficients of Gaussian components are indicated by different point sizes and shades of colour. The bigger the points are and the stronger the color is, the higher coefficient of the corresponding Gaussian components is. The topic centroids and their probabilities are detailed in Figure 4(c). We can observe that for topic T 3 named as "comp.graphics", the main components such as T 3:5, T 3:3, and T 3:0 are close to the cluster of red points, while T 2:8 and T 2:5 are close to sub-clusters of top words in    T 2.
To make a comprehensive comparison, we present the results of all models on generating topic "soc.religion.christian" in Table 5. It can be observed that our models can discover quite meaningful topics. We can also observe that the numbers of semantically irrelevant words of TMN and NVCTM are more than other models, which is consistent to the topic coherence results in Table 3.

Impact of Gaussian Mixture Numbers
We further study the impact of the number of Gaussian components. Table 6 presents the results of CRNTM GMD on 20NewsGroups when varying component numbers under 25 topics. We can observe that CRNTM GMD with more Gaussian com- ponents generally performs better than that with less ones, which demonstrates that a more sophisticated mixture possesses a stronger capacity of learning high quality topics. The best topic coherence and classification accuracy are obtained when the component number is set to 25, and a larger value may not further boost the model performance.

Conclusion
In this paper, we propose a Context Reinforced Neural Topic Model (CRNTM) to address the feature sparsity problem in short texts. By introducing a topic controller to the inference network, CRNTM infers the topic for each word in a narrow range. Besides, pre-trained word embeddings are incorporated with multivariate Gaussian distributions or Gaussian mixture distributions into our model to enrich the context information of short messages. To quantitatively validate the effectiveness of CRNTM, we conduct various experiments on two benchmark datasets in terms of perplexity, topic coherence, and text classification accuracy.
The results indicate that the proposed model largely improves the performance of topic modeling by enriching the context information effectively.