{'fraction_non_alphanumeric': 0.04255429162357808, 'fraction_numerical': 0.012874870734229577, 'mean_word_length': 4.773432835820896, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information

Non-negative matrix factorization (NMF) based topic modeling is widely used in natural language processing (NLP) to uncover hidden topics of short text documents. Usually, training a high-quality topic model requires large amount of textual data. In many real-world scenarios, customer textual data should be private and sensitive, precluding uploading to data centers. This paper proposes a Federated NMF (FedNMF) framework, which allows multiple clients to collaboratively train a high-quality NMF based topic model with locally stored data. However, standard federated learning will significantly undermine the performance of topic models in downstream tasks (e.g., text classification) when the data distribution over clients is heterogeneous. To alleviate this issue, we further propose FedNMF+MI, which simultaneously maximizes the mutual information (MI) between the count features of local texts and their topic weight vectors to mitigate the performance degradation. Experimental results show that our FedNMF+MI methods outperform Federated Latent Dirichlet Allocation (FedLDA) and the FedNMF without MI methods for short texts by a significant margin on both coherence score and classification F1 score.


I. INTRODUCTION
Topic models [1], [2] provide exploratory approaches to analyze large volumes of unlabeled text, which could discover clusters of tokens that co-occur often and organize large capacity of documents. They have many applications in text mining and analysis [3], [4]. In practice, most commonly used topic models can be grouped into two types: generative methods like Latent Dirichlet Allocation (LDA) [5], [6] and Non-negative Matrix Factorization (NMF) [7], [8], [9]. A few studies [10], [11] have proven that NMF outperforms LDA for short texts and is faster than the latter. Also, there are completely deterministic algorithms for its resolution [12], [13]. Therefore, NMF and its variants are commonly used to extract topics from short-text datasets [14], [15]. Topic models rely on large amount of textual data. Meanwhile, since data could be proprietary and sensitive, regulations such as the recently enforced European Union General Data Protection Regulation (GDPR) [16] may preclude uploading them to data centers and being utilized in the conventional centralized approach. Federated learning (FL) has been proposed to collaboratively train a machine learning model over decentralized data on clients [17], [18]. To address this issue for topic modeling, a few research investigate how to implement LDA in a federated manner [19], [20].
However, how to implement NMF based topic modeling in a federated manner is still underexplored. To fill this gap, we present a framework, called federated NMF (FedNMF), which enables multiple clients to jointly train a NMF based topic model. As shown in Table IV of our experiments (Section IV), simply implementing FedNMF with existing FL algorithms like FedAvg and FedAdam [17], [21], the performance of downstream text classification may degrade significantly for heterogeneous data over clients, in comparison with the centralized setting. To address this problem, we propose to maximize the mutual information [22], [23] between the original bag-of-word representations of documents and their topic weight vectors, which can be treated as highlevel representations of the raw documents. We introduce mutual information to topic modeling and show its effectiveness through experiments. The reasoning behind this technique is that topic weight vectors of the right factor matrix can be treated as a high-level representation of each document.
The contributions of this work are summarized as follows: • We propose a federated framework, FedNMF, which enables multiple clients to train a NMF based topic modeling with stochastic gradient descent (SGD). • To further boost the performance of FedNMF, we propose FedNMF+MI, which maximizes the mutual information between text features and its high-level representations. • We evaluate our framework on a few public short-text datasets to verify its effectiveness. Also we investigate its performance under varying federated settings.

II. RELATED WORK
Our work is related to three lines of research, which are federated learning, NMF based topic modeling and mutual information.
a) Federated Topic Modeling: FL, proposed by [17], aims to efficiently train a model by leveraging decentralized data over multiple clients. Since its inception, many algorithms such as FedAvg [17] and FedOpt [21] have been developed to reduce communication cost while retaining the performance. These algorithms usually proceed in two steps: (1.) synchronize the latest model weights from the server and train the model with local data; and (2.) after training, each client transmits the updated weights to the server for aggregation. These advanced FL algorithms are compatible to our FedNMF framework.
A few research has devoted to federated topic modeling. For example, [19] presented federated topic modeling that consists of novel techniques such as private Metropolis Hastings, topicwise normalization and heterogeneous model integration. [20] developed federated LDA that is based on a novel local differential privacy (LDP) [24] mechanism. These works mainly focus on the LDA based topic modeling. By contrast, in this paper we study the NMF based topic modeling under federated settings.
b) Mutual information: The mutual information between two random variables X and Y is the KL divergence between the joint and the product of marginals.
which we hope to estimate with samples from the joint probability P (X, Y ). Mutual information measures the inherent correlation between random variables (vectors). It has been widely used in unsupervised deep learning to improve representation learning for images, texts, etc [22], [25], [26], [27], [28]. Here we utilize the state-of-the-art mutual information estimator to enhance our FedNMF framework.

III. METHODOLOGY
Here we introduce our methodology. We use regular uppercase letters to denote matrices and boldface lowercase letters to denote vectors. For example, A is a m-by-n non-negative real matrix, whose element (i, j) is denoted by A i,j . More details on mathematical notations can be found in Table I. For a matrix A, NMF aims to find two find two nonnegative factor matrices W and H such that A ≈ WH. One straightforward way is to minimize the following 2 loss with respect to W and H.
For each document, it can be represented by a count (column) vector on top of bag-of-word representation. For documents on the i-th client, we denote the count feature matrix as A i . The concatenation of (A i , i = 1, . . . , K) is the matrix to be decomposed.
A. Federated NMF Figure 1 depicts how FedNMF factorizes multiple matrices distributed over different clients. In the federated learning framework, we use the stochastic gradient descent (SGD) algorithm to minimize the loss in Eq. (1). However, merely optimizing the squared loss on each client inside the FedAvg algorithm will lead to poor topic models, which is verified in the experiments in Section IV-E. Our intuition is to maximize the mutual information between the original textual features A i and features in the factor matrix H i . Directly estimate the mutual information is not tractable, and we bypass this difficulty utilizing the SMILE lower bound of mutual information [29] defined as where clip(v, l, u) = max(min(v, l), u), τ > 0 is a hyperparameter (fixed at 5.0 in this paper) and P, Q are the joint and marginal distributions of (x, y), respectively.
On a mini-batch (denoted as S, a set of column indices) of text documents on client i with batch size B, the SMILE estimator iŝ Maximizing the mutual information estimator in Eq. (2) forces the high-level features in the matrix H i to capture information in the original term-document matrix A i .
Plus the least square loss, the total loss on a certain minibatch S of client i is in which λ ≥ 0.0 controls the strength of MI regularizer. When training the model on the i-th client, the parameter θ of mutual The neural networks used in mutual information E Epochs of SGD on local clients estimator with parameter θ B Mini-batch size on local clients Algorithm 1 FedNMF+MI. The K clients are indexed by i; B is the local mini-batch size, E is the number of local epochs, and η is the learning rate. m ← max(C · K, 1)

5:
G t ←(random group of m clients) 6: for each client i ∈ G t in parallel do 7: 10: Agg is the aggregation rule, it can be weighted average (FedAvg) or adaptive optimization (FedOpt) 12: end for for batch b ∈ B do 5: i.e., optimizing θ by maximizing SMILE mutual information in Eq. (2) 8: end for 9: end for 10: return (W, θ) to server information estimator and matrix decomposition are separately learned, i.e., The complete pseudo-code of our FedNMF+MI framework is shown in Algorithm 1. When the MI regularizer weight λ = 0.0, this algorithm degenerates to FedNMF without MI, or plainly FedNMF.
In Algorithm 1, the Agg(·) is the aggregation function and it has many choices. The most basic one is weighted average that proceeds as follows This is the FedAvg algorithm, and our framework is also compatible to other aggregation algorithms like FedAdam, FedAdagrad and FedYogi. More details on these aggregation algorithms can be found in [21].

IV. EXPERIMENTS
In this section we implement our algorithms over a few public datasets and we compare the performance of our methods with the state-of-the-art FedLDA to show the effectiveness of our methods. All the models are implemented using PyTorch [30] and trained on NVIDIA Tesla V100 GPUs.

A. Datasets
The four real-world text datasets in the experiments correspond to various types of applications, i.e., Emails, News, Questions & Answers, Microblogs. a) Spam Ham: This data is a public set of labeled messages that have been collected for mobile phone spam research. It consists of 5,572 English, real and non-enconded messages, tagged according to being legitimate (ham) or spam. b) Tag.News: This data set is a part of the Tag News dataset3, which is composed of news, snippets and tweets [31]. The articles in the dataset belong to one of the following 7 categories: Business, Entertainment, Health, Sci&Tech, Sport, US and World. c) Rt Polarity: This dataset contains 5331 snippets for both positive and negative movie reviews, respectively. It was published and used in [32]. d) Stack Overflow: It is a collection of 20, 000 questions posted on StackOverflow.com, and each question is associated to tags like "html", "R", "python", etc. More details on this dataset can be found in [33]. Table II illustrates the basic statistics of datasets. In this table, '#docs', '#terms' and '#cats' represents the number of documents, terms and categories in each dataset, respectively. 'doc-length' is the average length of documents in each dataset. All these figures are computed after stop-words removal. In

B. Evaluation Metrics
Following [34] and [35], we utilize the word embedding (WE) topic coherence to evaluate the quality of topics. More importantly, we employ an external document classification [36], [37] to measure the performance of topic modeling methods. a) WE Topic Coherence: is defined as follows: where (w 1 , . . . , w N ) is the top N number of most probable words for each topic. Cosine(w i , w j ) is the cosine similarity of the word embeddings for words w i and w j , and here we use the 200-dimensional GloVe embeddings [38] trained on the Wikipedia 2014 and Gigaword 5 corpora. The average WE coherence score of all topics measures the quality of each topic model. b) Document Classification: Another popular way to evaluate the goodness of a topic model is to leverage the latent document representations for external tasks. In our experiments, we conduct text classification using the topic weights, i.e., the columns of H as features of documents.

C. Comparison Methods
Here are the list of topic modeling methods we implemented in our experiments from centralized methods to federated ones.
• Centralized Traditional Methods: LDA and NMF are conventional topic models for centralized textual data. • Centralized SGD-based Methods: We include SGDbased NMF methods (NMF+SGD and NMF+SGD+MI) in our experiments to verify the effectiveness of mutual information for the centralized NMF topic modeling. Essentially, for NMF+SGD, we optimize the least square loss of NMF with mini-batch SGD. An additional mutual information regularizer is included for NMF+SGD+MI. • Federated Topic Modeling: We implement three federated topic modeling approaches: FedLDA based on variational inference [19], [20], FedNMF with four implementation methods (FedAvg, FedAdagrad, FedYogi and FedAdam). We vary the value of λ to include or exclude the mutual information term, i.e., setting λ = 0 for plain FedNMF, and λ > 0 for FedNMF+MI.

D. Experimental Setup
In this experiment, we construct datasets for various clients in the same way as [39]. Specifically, we assume on every client training examples are drawn independently with class labels following a categorical distribution over l classes parameterized by a vector q with elements (q i ≥ 0, i ∈ [1, l] and q i = 1). We synthesize a population of non-identical clients by drawing q ∼ Dir(αp) from a Dirichlet distribution, where p is the label distribution of a certain dataset, and α controls the identicalness among clients. With α → ∞, all clients have identical distributions to p; with α → 0, on the other extreme, each client holds examples from only one label.
In our experiments, we use α to control the data heterogeneity among clients and vary client number K to create different FL settings. Throughout the whole experiments, we set K to the set {1, 10, 30, 50}, and to make our results comparable, we make each client to have equal number of documents, which is the total sample size divided by client number K. After the federated topic modeling, we aggregate the topic weight vectors of all documents, randomly split it into train set (80%) and test set (20%), and train a support vector machine (SVM) classifier on top of them to yield the macro F1 score and accuracy. We set the number of topics k to four values 20, 50, 100 and 200 for all datasets to obtain comprehensive results. For the federated topic modeling approaches, we set the fraction of participants for each round, C = 0.2. At each iteration, we tune the number of epochs for local SGD training from the set {10, 20, 30, 40}, and local batch size from the set {16, 32, 64, 128}, the learning rate from the set {1.e − 2, 2.e − 2, 5.e − 2, 1.e − 1}, the weight λ from the set {0, 0.01, 0.05, 0.1, 0.5}. The hyper-parameters in FedAdagrad, FedYogi and FedAdam are set as default in [21]. The neural networks T θ used for mutual information is a 2-layered feed-forward networks with 32 and 256 neurons in the 2 hidden layers, respectively.

E. Results and Analysis
Table III and IV illustrate the coherence score and text classification metrics of various topic modeling approaches over four datasets. The reported coherence scores, F1 scores and accuracy values are averaged over three topic numbers k = 50, 100 and 200. The FL environment is set as α = 1.0 and K = 10. Here are the main observations: 1.) By comparing traditional LDA and NMF, we find that NMF outperforms LDA on all datasets, and these documents are short in general, smaller than 30 by length. Therefore, NMF usually outperforms LDA on short documents. 2.) Mutual information benefits NMF topic modeling under both centralized and federated settings. This is shown in both coherence score and classification. For instance, in Table IV, NMF+SGD+MI consistantly yields better F1 scores than NMF+SGD in centralized learning over four datasets, especially on Rt polarity, which presents a 15% margin. This exhibits the effectiveness of mutual information in NMF based topic modeling. 3.) Among the federated topic models, FedNMF+MI methods (including FedAvg+MI, FedAdagrad+MI, FedYogi+MI, and FedAdam+MI) produce better results than FedLDA on all four datasets, though FedNMF without MI (FedAvg, FedAdagrad, FedYogi and FedAdam) perform slightly better than FedLDA. a) Classification over topic number: The text classification is closely related to the number of topics and Figure 2 presents the classification accuracy versus topic number on three datasets. The FL setting is α = 1 and K = 10, solid lines represent FedNMF+MI methods while dotted lines for FedNMF without MI methods. We utilize red, blue, green and purple to color four different methods: FedAvg, FedAdagrad, FedAdam and FedYogi. Throughout this section, we apply the same plotting scheme. In general, as the increase of topic number K, the text classification accuracy rises, which is due to the rapid growth of model capacity. The solid lines are consistently above the dotted lines of the same color, which shows the effectiveness of MI in the improvement of downstream classification task.  FedYogi respectively. The solid lines tends to converge faster than the dotted line of the same color, which means that MI makes the FedNMF converges faster. c) Topic Examples Evaluation: To qualitatively illustrate the high-quality topics generated by federated learning approaches, Table V presents the examples of topic words yielded by FedLDA, FedAvg, FedAdam, FedAvg+MI and FedAdam+MI algorithms on the Stack Overflow dataset. This table exhibits that FedLDA tends to generate some repetitive topics with repeated words, such as "bash script" and "command line", and although the topics of FedAdam and FedAvg seem diverse, they are less informative and less coherent. The topic quality of FedAdam+MI and FedAvg+MI is apparently higher than others. This also illustrates the effectiveness of MI for FedNMF algorithms. d) Varying α and Client Number: We also study the effect of data heterogeneity and client number on FedNMF+MI algorithms. Figure 4 displays the performance of FedAvg+MI and FedAdam+MI over varying values of α and client number K. Across two datasets, the general trend is that FedNMF performs better as α increases and performs worse as client number K increases. This is because the data heterogeneity across clients decreases as α increases, so it makes it easier for federated methods (both FedNMF and FedNMF+MI) to yield well-trained topic models. As the client number K increases, the data is more scattered, so it becomes harder for FedNMF(+MI) algorithms to extract good topics.

V. CONCLUSION
In this paper, we propose a framework, FedNMF, for NMF based federated topic modeling methods, which can yield   high-quality topics when the documents are locally stored. To mitigate the performance degradation caused by data heterogeneity across clients, we propose the FedNMF+MI framework, which further maximizes the mutual information between the input text count features and topic weights. This sheds light on the potential use of mutual information in NMF based topic modeling. As the increasing awareness on data privacy, our FedNMF+MI algorithms can be helpful in many applications such as the decentralized short texts analysis and short document content mining.