{'fraction_non_alphanumeric': 0.029571360938125932, 'fraction_numerical': 0.00961433409810991, 'mean_word_length': 5.0873420527599205, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TopicNet: Making Additive Regularisation for Topic Modelling Accessible

This paper introduces TopicNet, a new Python module for topic modeling. This package, distributed under the MIT license, focuses on bringing additive regularization topic modelling (ARTM) to non-specialists using a general-purpose high-level language. The module features include powerful model visualization techniques, various training strategies, semi-automated model selection, support for user-defined goal metrics, and a modular approach to topic model training. Source code and documentation are available at https://github.com/machine-intelligence-laboratory/TopicNet


Introduction
Topic modelling is a method to extract hidden word probability distributions called topics from text corpora. Being primarily introduced to find latent topics in text documents, topic models have proven to be relevant in a wide range of contexts . Over the time topic models have primarily been used for two purposes: either providing embeddings over collections of text for various recommendation and search purposes or giving a cohesive clustering of data for purposes of data mining and extracting information about the collection structure. The first problem requires the researcher to tune their models to perform well on labelled datasets, thus, reducing the task to classification problem using obtained from topic model document embeddings as feature vectors. This makes topic models compete with various machine learning and deep learning algorithms in the field of this classical task. Topic modelling does not necessarily show the best results in terms of classification metrics, but it has an edge when it comes to interpretability. Each topic can be interpreted as a probability distribution over words in that topic, and each coordinate in a document embedding has a meaning as a probability for that document to contain a certain topic in it. This property of topic models makes them better suited for the fields of AI where clarity of prediction matters or where one needs to correct undesirable biases introduced by the data into the model. It should be noted that topic modelling is still a valuable tool in natural language processing (NLP) researcher arsenal. Currently, there is an ongoing work to extend stateof-the-art deep learning models to incorporate topical importance of words for text summarization (Narayan et al., 2018;Wang et al., 2018;Lebanoff et al., 2019) and song lyrics understanding (Choi et al., 2015;Fell et al., 2019;Watanabe et al., 2018). One can see that topic modelling and its numerous variations continues to be a workhorse in many NLP projects.
The second purpose -collection structure retrieval, however, is somehow unique to the field of topic modelling. Given that the researcher not necessarily wants to optimise the classification metric, but merely searches for answers about the collection structure and nature. This approach, for example, has been adopted in the fields of biology (Liu et al., 2016;Funnell et al., 2019) and humanities Antons et al., 2019). It provides an invaluable insight into big data that otherwise might have been missed by the researcher. Jordan Boyd-Graber, Yuening Hu and David Mimno in their monograph  show, that designing new topic model is a difficult enterprise and identify accessibility as the most important unsolved problem in topic modelling: "the primary research challenge of topic models is... to make them more accessible". The same year another publication (Lee et al., 2017) argued, that existing topic models do not provide non-expert users with direct means to alter topic features perceived by these users to be bad. The study suggested several improvements for existing topic modelling user experience, concentrating on topic visualisation for the users. The article from another group (Agrawal et al., 2018) demonstrate that existing popular topic models are failing to provide good results when used "off-the-shelf" with their default parameters. To facilitate the research on mining latent topics from texts and text-like collections, we present a novel software package called TopicNet 1 . This software package contributes to the flexibility of topic model design and provides powerful out-of-the-box experience boosting the accessibility of topic modelling for the general audience.
1 Source code can be downloaded from github.com/

Related Work
Existing topic modelling algorithms originate from latent semantic analysis approach -LSA (Deerwester et al., 1990), which is, in the essence, a singular value decomposition with the lowest possible rank for document×term matrix. Later, a probabilistic approach took the main place in the topic modelling field. A probabilistic topical model can be considered as a black box that receives a text document collection at the input and produces two families of distributions at the output: the term probabilities for each topics φ wt = p(w | t) and the topic probabilities for each document θ td = p(t | d).
The term×topic matrix Φ and topic×document matrix Θ are model parameters to be found during the training. The development of probabilistic topic modelling started with two fundamental models: probabilistic Latent Semantic Analysis -pLSA (Hofmann, 1999) and Latent Dirichlet Allocation -LDA (Blei et al., 2003). LDA adds Dirichlet prior to pLSA: the generative LDA model draws termtopic and topic-document distributions from prior Dirichlet distributions, each with its own concentration parameter. While writing a document, authors usually use only handful of topics per each text. This sparsity property is reflected in concentration parameter. Varying value of this parameter, one can obtain distributions where only a few topics have high probability to be in a document with other topic probabilities being small. Thus, the sparsity of Dirichlet distributions is the probabilistic tool that encodes this intuition. However, according to the article (Wallach et al., 2009), LDA requires extensive hyperparameter optimization toward producing good results. Over the past years, hundreds of pLSA and LDA model extensions have emerged, each taking into account the various problem-specific data features and providing desired solution properties. Starting with LDA model, Bayesian learning is the de facto standard in topic modelling. In this approach, one first describes the probabilistic generative model of data, specifies prior distributions of the model parameters, and then uses Bayesian inference to obtain the parameter posterior distributions. Bayesian inference method requires unique derivation and, consequently, unique implementation for each new model. Additive regularization for topic modelling -ARTM (Vorontsov, 2014;Kochedykov et al., 2017) is a non-Bayesian multi-objective approach. It is based on the maximization of the log-likelihood together with a weighted sum of regularization criteria. Many of the well-known Bayesian topic models can be reformulated as a regularization over the PLSA model. After reformulation, they usually become much easier to understand and implement. The ARTM approach allows combining topic models simply by adding the regularizers. This gave rise to the modular technology for topic modelling implemented in the BigARTM 2 open-source software (Vorontsov et al., 2015c). Given a generative model and data, inference must be executed for extracting probabilistic topic-depending distributions. There are many inference algorithms: expectationmaximization (EM) algorithm, Gibbs sampling, variational inference, gradient descent and message passing. In ARTM the regularized EM-algorithm is used to learn model parameters. The similarities between each of the algorithms were noted before in (Asuncion et al., 2009). Gensim (Řehůřek and Sojka, 2010) is the most popular NLP framework for Topic Modeling. It implements several popular models such as LSA, pLSA, LDA, Hierarchical Latent Dirichlet Allocation (HLDA) and their derivatives. This framework also implements a coherence metric to improve topics from previously mentioned models as done in the article (Röder et al., 2015). This framework is written in Python and optimized for huge document corpora. Stanford Topic Modelling Toolbox -TMT (Ramage et al., 2009), based on Scala, has LDA, Labelled LDA and PLDA models available for training. Stanford TMT also includes user-friendly interaction with Excel, allowing one to load data from Excel cells and generate rich output for tracking word usage across topics, time and other groupings of data. MALLET (McCallum, 2002) is an essential framework for any researcher in Topic Modelling field, which is implemented in Java. This toolkit contains efficient, samplingbased implementations of LDA, Pachinko Allocation (Li and McCallum, 2006), and HLDA models. The framework provides advanced LDA models and often used for online services (Pol et al., 2017). A Library of Short Text Topic Modelling -STTM (Qiang et al., 2018) is the Java framework that extends the range of available open-source modelling methods and integrates the state-of-the-art models of short text topic modelling algorithms. Primarily it aims at distilling meaningful topics from short texts, thus many high-performance models such as Dirichlet Multinomial Mixture DMM (Yin and Wang, 2014), Word Network Topic Model WNTM (Zuo et al., 2016b), Pseudo-Document-Based Topic Model PTM (Zuo et al., 2016a) and Self-Aggregation-Based Topic Model SATM (Quan et al., 2015) present in this framework. Some long-text topic models such as LDA and Latent Feature Model with LDA (Nguyen et al., 2015) are provided by STTM as well. Familia (Jiang et al., 2018) is a framework that implements various topic models, including, but not limited to, chief LDA and Supervised LDA: Topics Over Time TOT (Wang and McCallum, 2006), Bilingual Topic Model (Gao et al., 2011), Location-Aware Topic Model LATM (Wang et al., 2007) and some more, using Gibbs sampling as a mathematical engine. To the best of our knowledge, despite authors claim that Familia provides the ability to "design their own topic models", we found no such evidence in the repository of the project (Lian, 2019). As identified before, one of the main challenges in the field of topic modelling is the accessibility of the models to the general public. Each previously mentioned framework succeeded in closing that gap at the time of their release, but due to the nature of the Bayesian approach, the most popular frameworks (Gensim, MALLET) provide outdated models. The other challenge is building complex, tractable and multi-objective topic models from scratch. While the basic model can be implemented in a reasonable amount of time, improving it remains both a time-consuming and an error-prone task (Jiang et al., 2018) With our work, we aim not only to close the gap between novel models and popular models as our predecessors do but also provide a tool that will allow anyone to construct new types of topic models. Due to ARTM formalism, Top-icNet offers natural language processing community access to Python-based multimodal topic modelling that supports large documents and huge corpora.

Underlying Technologies
Having carefully considered all mentioned approaches to topic modelling, we decided to build a framework that will allow a better user experience with the BigARTM library. Below, we discuss in greater detail pluses of the BigARTM library python API and its minuses that we wanted to deal with in the TopicNet.

BigARTM Strengths
BigARTM is a fast and flexible library for topic modeling (Frei and Apishev, 2016), based on Additive Regularization of Topic Models (ARTM) formalism (Vorontsov, 2014). The idea behind ARTM is to replace likelihood with regularized likelihood and optimize this functional using modified EM-algorithm. The regularization serves two purposes. First, it ensures robustness and limits solution area, reducing the instability of inference. Second, each regularization term is used to pursue different solution characteristics such as sparsity, diversity, coherence or to take extra information into account. As an example of dealing with auxiliary information, BigARTM makes it very easy to include document metadata (e.g. authors, timestamps, tags, and n-grams) in a single model. This is because the likelihood of each additional modality could be considered as a regularizer applied to the topic model over words. Works that take advantage of ARTM's and BigARTM's flexibility include: exploratory search quality improvement (Yanina et al., 2018), learning interpretable topical word embeddings through word network topic model (Potapenko et al., 2017), hierarchical topic modelling (Chirkova and Vorontsov, 2016), multi-label text categorization (Vorontsov et al., 2015b), improving topics for document vector representation in text regression problems (Sokolov and Bogolubsky, 2015), finding rare ethnically relevant topics in social media (Apishev et al., 2016a;Apishev et al., 2016b), incorporating language features (Popov et al., 2019), topic selection through entropy regularization (Vorontsov et al., 2015a), improving topics through text segmentation (Skachkov and Vorontsov, 2018), directly improving topic coherence (Mavrin et al., 2018), surpassing the bag-of-words hypothesis by using a new intra-text coherence measure (Alekseev et al., 2018). The review (Kochedykov et al., 2017) shows how Bayesian topic models can be re-formulated in a much simpler way from the ARTM point of view, including multimodal, multilingual, temporal, hierarchical, graph-based, and short-text topic models. The development of both ARTM theory and BigARTM software is still ongoing. Many of the existing widely used regularizers were contributed by community.
To the best of our knowledge, among other topic modelling frameworks, only Familia offers comparable flexibility. However, the ability to construct a custom topic model or even to train one on a given corpus is absent from the open-source release of Familia (Lian, 2019).

BigARTM Flaws
Given a precise specification of a regularized model, BigARTM can infer it's parameters in a very fast, scalable and efficient way. However, it is unclear where such specification comes from. While the number of topics is an unresolved question in many topic modelling frameworks, the issue is further complicated by the ability to combine many different regularizers, each having an unknown individualised regularization coefficient. BigARTM offers virtually no guidance regarding the selection of regularizers and their structural parameters. An additional factor contributing to the high entry barrier is somewhat inconvenient and inconsistent API in BigARTM library. This is a natural result of implementing new functionality before "best practices" of its usage are established and inability to change API afterwards due to backwards compatibility concerns. Therefore, as applications of BigARTM were becoming more diverse and the algorithms were gradually refined, the high-level interface of BigARTM was getting less well-suited for "best practices". Another shortcoming of BigARTM is the difficulty in extending it. From a technical point of view, BigARTM library consists of a core written in C++ and several Python wrapper classes. Low-level C++ routines are multithreaded and highly optimized, giving BigARTM an edge in performance. At the same time, it makes any modification of low-level functionality challenging. Meanwhile, the highlevel API does not always offer enough flexibility to, for example, experiment with new custom regularizers.

Project Vision
The main motivation of the TopicNet is to close the gap between non-experts and power users. It does not mean that both groups will use the library in the same way; rather, it means that both groups should be able to communicate with each other. We formulated the following requirements necessary to achieve these aims: • Modularity: it should be possible to use just a small part of TopicNet functionality as "plugin" inside an independently existing project. The reason behind this was twofold. The first point was that it should make the adoption easier: power users are not forced to dramatically change their existing projects to start getting benefit from TopicNet. The second point is related to the building an open-source community around Topic-Net: modular open source projects are more welcoming to the contributors.
• Visualisation tools: the library should provide readyto-use powerful visualisation tools. Such tools play an important role in error analysis and might be helpful for downstream tasks (e.g. exploratory search). Following our previous requirement, this module should be as stand-alone as possible; ideally, it would allow the community to incorporate best practices from literature into TopicNet.
• Concision: the library should remove the low-level details, freeing more time for substantial problems. The reason behind this was threefold. The first point was the need to improve the artistic process of constructing a target function for the task at hand. The second point was the adherence to the "convention over configuration" philosophy: by reducing the amount of explicitly declared things and providing sensible defaults, we can enforce "best practices" such as automatically saving trained models or storing data batches separately for different datasets. The final point is readability: when working with concise and uniform code, the user can see more code lines on the screen which aids better reading and understanding the experiment. As a result, it is much simpler to share, review and debug finished experiments.
• Work out of the box: the library should include some pre-coded training pipelines which are ready-to-use and produce good results. Moreover, these pipelines should absorb the best-known approaches for as many modelling tasks as possible. The positive "out of the box" experience is vital for involving novice users, whereas power users may deliver their experience through pre-coded pipelines.

Code Design
The TopicNet consists of two large modules: viewers and cooking machine. The purpose of Viewers module is to provide powerful visualisation tools. The design adheres to the Unix philosophy: each viewer has a limited area of responsibility and returns the result as a JSON-convertible object by default. Consequently, the module has a high degree of composability and modularity while still having convenience methods returning pandas.DataFrame or rendered HTML. Examples of the viewers output, like the TopTokensViewer and the DocumentClusterViewer, can be seen in Figures 1,  2 respectively. The Cooking Machine module contains all modelling toolbox, embodied in the semi-hierarchical structure of the main modelling classes. These classes are responsible for building and training model of the given structure, for selecting models according to various constraints and for saving, loading and logging through the modelling process. Following our "convention over configuration" principle, we enforce some assumptions on which kind of experiments TopicNet supports. We hold that model training pipeline could be represented as a tree. Each node is a topic model and directed edges represent parent-children relationship, such as "model Y was obtained from model X with the transformation T XY ". We restrict allowed transformations by linking them to their depth in the experiment tree: we require that each edge of the same level describe the same transformation aside from a set of individualised parameters.
A non-exhaustive list of such transformations: • Applying a regularizer with arbitrary regularization coefficient or changing parameters of the existing regularizer • Training a model for several iterations • Adding topics inferred on a different corpus to the model The Experiment class is responsible for storing, logging and maintaining this structure. The transformations are connected to the instances of Cube class. Each Cube acts as a blueprint for all model transformations performed on the current stage of the experiment. In a way, a training pipeline is several cubes stacked together sequentially. Cube is responsible for two essential functions. The first use is specification: during the initialization stage, cube converts user-defined parameters into a multidimensional search space. The second use is alteration: given a point in search space and a topic model, cube alters one or many model hyperparameters. That way it acts as an incubator for models, which is reflected in the class name. Scheme of the training process with two cubes, applied to the model, can be seen in Figure 3. Taken together, Experiment and Cube classes make logging and complex training pipelines more concise and accessible. To capitalise on this decision, we implemented a config parser module which allows specifying complex training pipelines as a plaintext config file in YAML format.
Another key area described as very verbose and confusing is model selection. In real experiments, not every model has descendants; most models are rejected based on some criteria. Aside from expensive manual inspection of top words and top documents, other conventional criteria include perplexity and coherence. BigARTM library adds several other metrics such as sparsity, purity and contrast .
To reduce the burden of manual inspection, we implemented a simple domain-specific language for model selection (see Figure 4 for example). This language simplifies the selection task by conveniently leveraging various metrics.

Comparison with Related Work/Benchmarks
To compare the performance of the TopicNet with other frameworks, we chose a few essential areas. First of all, we wanted to check that we do not loose much time on training a single model or spend too many resources on it compared to the other frameworks. The other test would be measuring topic interpretability and diversity for each model. To measure interpretability we are using Umass coherence provided by Palmetto web service (Röder et al., 2015) as it is shown to correlate with the desired property of the topics (Mimno et al., 2011). The Jaccard metric is used to estimate topic diversity across the model topics.  At the first stage, regularizer with parameter τ taking values in some range {τ 1 , τ 2 , τ 3 } is applied. Best models after the first stage are Model 1 and Model 2 -so Model 3 is not taking part in the training process anymore. The second stage is connected with another regularizer with parameter ξ taking values in range {ξ 1 , ξ 2 }. As a result of this stage, two descendant models of Model 1 and two descendant models of Model 2 are obtained.

Resources Usage
Important library feature, from the scope of accessibility, is the usage of computer resources during the model training. If we want to engage a broader community such as students or aim at production applications of our models the framework should be as fast and lightweight as possible. We chose the NIPS dataset to run a performance test (McCallum, 1996). even single processor version beats other frameworks in terms of performance. Meaning that TopicNet overhead over BigARTM did not decrease its performance too drastically.

Model Quality
Following our library developing aims, we look for a good out of the box experience for a new user. To simulate such conditions we train TopicNet model from ARTM baseline recipe and compare it with models from other frameworks. For each framework in our comparative experiment, one model is trained with default or built-in parameters. LDA and all STTM models had the number of topics equal 20, for TopicNet model we set 19 topics and one "background" topic, which has a special set of regularizers to collect polythematic documents. TopicNet model was constructed from the baseline training recipe without any parameter tuning. As an example of how to use TopicNet we provide the code needed to train baseline topic model discussed above 5. As a demonstration dataset, we chose a well known 20 newsgroups dataset. To make a fair model comparison we used Umass coherence score provided on Palmetto Demo website 3 . To assess the diversity of the topics provided by each model we used the Jaccard similarity coefficient. Both measures were calculated on the top ten tokens for each topic and every model.  Figure 4: This expression returns three models which are in the top 5% according to contrast, has acceptable perplexity and as sparse as possible. SparsityPhiScore stands for the fraction of zeros in φ wt = p(w | t) distribution.  As one can see from the above table 2, TopicNet model was a second-best model in terms of interpretability and the best model in terms of topic diversity. Which can be accounted to a Decorrelation regularization implemented in the ARTM baseline recipe. Together with a custom score provided by the TopicNet library, Decorrelation regularizer allows to strike a balance between perplexity minimization and topic interpretability. Further information on a topic coherence can be seen in Figure 6.

Conclusion
In this paper, we propose a configurable and fast framework for topic modelling and demonstrate its advantages over its competitors. TopicNet provides extensive functionalities such as building models from scratch, rich model customization and a possibility to fine-tune any previously constructed models.
The library provides modelling recipes capturing bestknown practices of building an ARTM model for a certain task. As demonstrated in the previous chapter, an ARTM model is capable of beating most popular models in terms of providing cohesive and diverse topics. With TopicNet, Figure 6: Topic coherence distribution.
engineers can customize it further by applying custom regularizers and finding best hyperparameter values for multicriteria training scenario. Taken together, this allows the user to build their own type of topic model. Aside from tools for flexible and quick model development, we enable users to control their model quality. Topic-Net provides model evaluation through a variety of built-in scores and the ability to add custom scores, which are used to track model characteristics during training. The library supports a variety of visualisation tools both conventional, such as top tokens and top documents viewers, and experimental ones. Moreover, our framework is delivered as an open-source project, with potential for further extension. The design philosophy behind viewers and cooking machine modules allow open source community to incorporate new developments and discoveries into TopicNet library. We hope that our framework will be equally valuable to software engineers and digital humanities researchers.

Acknowledgments
We are thankful to our colleagues Rosa Aisina, Murat Apishev, Ilya Zharikov, Filipp Nikitin, Artem Popov, Vera Shishkina, and Oleksandr Frei who provided the expertise that greatly assisted the project, although they may not agree with all of the project implementations presented in this paper.

References
Agrawal, A., Fu, W., and Menzies, T. (2018). What is wrong with topic modeling? and how to fix it using search-based software engineering. Information and Software Technology, 98:74-88. Alekseev, V., Bulatov, V., . Intratext coherence as a measure of topic models' interpretability. In Computational Linguistics and Intellec-