{'fraction_non_alphanumeric': 0.027312413715906123, 'fraction_numerical': 0.007163911794336032, 'mean_word_length': 5.1066302118933695, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semantic flow in language networks

In this study we propose a framework to characterize documents based on their semantic flow. The proposed framework encompasses a network-based model that connected sentences based on their semantic similarity. Semantic fields are detected using standard community detection methods. as the story unfolds, transitions between semantic fields are represent in Markov networks, which in turned are characterized via network motifs (subgraphs). Here we show that the proposed framework can be used to classify books according to their style and publication dates. Remarkably, even without a systematic optimization of parameters, philosophy and investigative books were discriminated with an accuracy rate of 92.5%. Because this model captures semantic features of texts, it could be used as an additional feature in traditional network-based models of texts that capture only syntactical/stylistic information, as it is the case of word adjacency (co-occurrence) networks.


Introduction
In the last few years, several interesting findings have been reported by studies using network science to model language [1,2,3,4]. Network-based models have been used e.g. to address the authorship recognition problem, where the structure of the networks  Figure 1: Illustration of process of construction a text from a network of a ideas. Each node represents an idea in this example and edges represent the relationship (similarity) of ideas. A written text can be seen as a walk on this network (see e.g. [15]). In the example, the following ideas are produced in the text: "A, B, C, K, M, L, 0". analysis of how authors navigate the high-dimensional semantic relationships to generate a linear stream of words, sentences or paragraphs. In [20], a mesoscopic representation of networks was proposed. The authors used as a semantic, meaningful block a set of consecutive paragraphs. The semantic blocks were connected according to a lexical similarity index. The model aimed at combining a networked representation with a idea of semantic sequence obtained when reading a document. Even though some interesting patterns were found, the concept of semantic fields were not clear, as no semantic community structure arises from mesoscopic networks. The problem of linearization of a network structure was studied in [15]. A systematic analysis of the efficiency of several random walks in different topologies was probed. The efficiency was probed in a twofold manner: (i) the efficiency in transmitting the projected network; and (ii) the efficiency in recovering the original network.
In [21], the authors explored the efficiency of navigating a idea space, by varying network topologies and exploration strategies.
In the current paper, we take the view that authors write documents by applying a linearization process to the original network of ideas, as shown in the procedure illustrated in Figure 1. Upon analyzing the flow of ideas with the adopted network-based framework, we show that features extracted from the networks can be employed to characterize and classify texts. More specifically, we defined the network of ideas as a network of sentences linked by semantic similarity. Semantic fields of similar sentences (nodes) were identified via network community detection. These fields (network communities) were then used to characterize the dynamics of authors' choices in moving from field to field as the story unfolds. Using a stochastic Markov model to represent the dynamics of choices of semantic fields performed by the author along the text, we showed, as a proof of principle, that the adopted representation can retrieve textual features including style (publication epoch) and complexity.

Research Questions
The main objective is to answer the following research questions: is there any patterns of semantic flow in stories? Are these patterns related to textual characteristics? To address 4 these questions, we use sentence networks to represent the semantic flow of ideas in texts.
Such networks are summarized using a high-level representation based on the relationship between communities extracted from the sentence networks. Using this representation, we show that motifs extracted from such a high-level representation can be used to classify texts according to the style in which authors unfolds their stories. We argue that the obtained results suggest that the proposed high-level view of a text network could be further probed in other Natural Language Processing classification tasks. This paper is organized as follows. Section 3 presents some concepts used in the proposed methods along with the method/framework itself. Section 4 presents the details of the experiments and results with a thorough discussion. Finally, in Section 5 we present some perspectives and insights for further works.

Materials and Methods
This study can be divided in two parts. In the first step, we identify the semantic clusters (fields) of the story. Differently from the analysis of short texts, where semantic groups can be identified mostly by identifying paragraphs, in long texts -the focus of this studythe identification of semantic clusters is more challenging because semantic topics might not be organized in consecutive sentences/paragraphs owing to the linearization process blueillustrated in Figure 1. In other words, the process of obtaining semantic clusters can be understood as the reverse operation depicted in Figure 1.
In order to identify semantic clusters from the text, we first create a network of sentences for each document, where sentences are linked if the similarity between them is above a given threshold. The obtained network is then analyzed via community detection methods, where groups of densely connected sentences are identified and considered as semantic clusters. A qualitative analysis of the obtained communities suggested that most of the largest communities are in fact related to a specific subtopic approached in the text (see details in Section 3.3). This idea relating semantic fields and network communities has also been used to construct automatic summarization systems [22].
In the second step of this study, we investigate the semantic flow of ideas developed by authors while unfolding their stories. We consider each community found as a semantic cluster, and as the story unfolds (one sentence after another), we analyze the community labels of the adjacent sentences to create a Markov chain, where each state represents a community and transitions are given by the text dynamics. Once the Markov chain representing the transitions of semantic clusters is obtained, the text is characterized by finding and counting different chain motifs. Such a characterization is then used to classify texts according to the semantic flow as revealed by sentences membership to different network communities.
The main objective of this work is to provide a framework to analyze and verify whether the semantic flow in texts can be used to characterize documents. Because the framework encompasses some steps, several alternatives could be probed in each step. We decided not to conduct a systematic analysis of combination of methods (and parameters) owing to the complexity of such analysis. A systematic study of the parameters and methods optimizing the proposed framework is intended to be conducted as a future work.
In Figure 2 we show a representation of the framework proposed to analyze stories. In the next section, we detail each of steps used in this framework.

Word and Sentence Embeddings
Usually any vector representation of words is known as a word embedding. However, since the creation of neural word embeddings [23], the term is mostly used to name those approaches based on neural network representations. The word embedding model proposed in [23] aimed at classifying texts based on raw text input. Thus, the classification does not require that textual features as input. Typically, word embeddings are dense vectors that are learned for a specific vocabulary, with the objective of addressing some task.
A typical task addressed with word embeddings is the language modeling problem, which aims at learning a probability function describing the sequence of words in a language. More recently, this same vector representation has been used in more complex models, with the objective of addressing several Natural Language Processing tasks simultaneously, including POS tagging, name entity recognition, semantic role labeling and others [24,25]. Despite

Entrada
Projeção Saída its relative success in the above mentioned tasks, the adopted embeddings could not be used in general purpose applications [24,25]. In order to allow the use of embeddings in wider contexts, the Word2Vec representation was proposed [26,27].
The Word2Vec is a neural model proposed to learn a dense, high-quality representation that is able to capture both syntactical and semantical language properties. As a consequence, vectors representing words conveying the same meaning are close in the considered space. An interesting property of the Word2Vec technique is the compositionality, which allows that large information blocks (e.g. sentences) can be represented by combining the representation of the vector representing the words in the sentence. Other interesting property is the ability to combine embeddings in a intuitive fashion [26,28]. For example, using the Word2Vec technique, the following relationship can be obtained: vector("King") -vector("Man") + vector("Woman") vector("Queen"). (1) The Word2Vec model is a robust, general-purpose neural representation that has been widely used in several Natural Language Processing tasks, including machine translation [29], summarization [30,31], sentiment analysis [32] and others. Given the success of the this model and the possibility of composition in different scenarios (sentiment analysis and sense disambiguation) [32,33,34], in the current study we used a representation of sentences based on the Word2Vec. More specifically, here the embedding s of a sentence s is represented by the average embedding of the words in s: where w i is the embedding of the i-th word in s and ω(s) is the total number of words in s.
The word embedding technique used here was obtained with the Word2Vec method (skip-gram). The training phase used the Google News corpus [27,26]. According to [27,26], the parameters of the method are optimized in the context of semantical similarity task. The combination of embeddings to represent a sentence in equation 2 could also be performed by summing individual embeddings. However, it has been shown that there is no significant difference when sentence embeddings are used to construct a network of sentence 8 similarity [35]. We note that some words are removed from this analysis. This includes stopwords (e.g. articles and prepositions) and words with no embeddings in the Google News corpus. Thus, whenever a sentence contains only words with no available embeddings, it is removed from the analysis.

Modeling sentence embeddings into complex networks
This step corresponds to the reverse process illustrated in Figure 1. In other words, a network representing the relationship between ideas is created from the text. The construction of networks from vector structures has been explored in recent works. In [36], the authors present such a transformation as a framework in complex systems analysis. The transformation of vector structures into networks has also been used in the context of text analysis [35,37]. The creation of a complex network from Word2Vec was proposed using a twofold approach. The d-proximity technique links all nodes whose distance from the refence node is lower than d. The second technique is the k-NN approach, which links all k nearest nodes to the reference node. In the same line, [35] created a network based on word embeddings. However, the authors aimed at creating a network that takes into account the sense of words to solve ambiguities. Each occurrence of an ambiguous words was modelled as a node in the network. Nodes were represented by a vector combining the embeddings of the words in the context. Two occurrences of an ambiguous were then connected whenever the respective embeddings were similar. In other words, two ambiguous words were linked if they appeared in similar contexts.
In the currrent study, sentences were connected according to the k-NN technique, as suggested by other works [37]. Each sentence is represented as a vector according to equation 2. The value of k in the main experiments were chosen to allow that each network is composed of a single connected component. In particular, the lowest k allowing the creation of a connected network was used for each book.

Community detection
The next step in the proposed framework concerns the detection of semantic fields, i.e.
the communities in the network of sentences. A recurrent phenomena in several complex networks is the existence of communities, i.e. groups of strongly connected nodes. Similarly to other network measurements, the detection of communities gives important information regarding the organization of networks. Communities are present in different networks including in biological, social and information networks [38].
A well-known measure to quantify the quality of partitions in complex networks is the modularity [39,40]. This measure compares the obtained partition with a null model, i.e. a network with similar properties but with no community structure. Several algorithms have been proposed to address the community detection problem via optimization of the modularity. In the main experiments we used the the Louvain method [41] to identify communities. The main advantage of this method is its computational efficiency, which has allowed its use in several contexts [37,42]. Another advantage associated to this algorithm is that no additional parameters are required to optimize the modularity.
In the proposed network representation, communities represent groups of interconnected sentences about a given topic. Because the k-NN construction allows nodes to be connected to other close nodes and, considering the Word2Vec an efficient semantic representation, the linking strategy allows the creation of dense clusters of semantically related sentences.
This idea of semantic clusters has also been explored via community detection in similar works [22,35,37]. For example, using networks built at the word level, the groups detected in [37] were found to represent large cities, professions and others topics. In [35], the obtained groups were found to represent words conveying the same sense.
In order to illustrate the process of obtaining semantic communities, we performed an analysis of the obtained communities in the book "Alice's Adventures in Wonderland", by Lewis Carroll. We summarize below the main topics approaches in some of the communities obtained by the Louvain algorithm: 1. Community A: this community includes sentences mentioning animals (e.g. "pet", "cat", "mouse" and "dog"). This community also includes dialogues between Alice and animals. "Cat" is the main character in this community.
2. Community B : this community includes words sentiment words expressed via speeches.
3. Community C : this community includes several adverbs related to Alice's actions.
4. Community D: this community includes words related to sentiments such as anger, tranquility and peacefulness.

5.
Community E : this community is most related to the word "soup".
6. Community F : this community is related to geographical locations, including countries and cities (Australia, Rome and New Zealand). Interestingly, this community also included the word "Cricket", a prominent sport in Australia.
7. Community G: this community included mostly sentences referring to "Dormouse", one of the main characters in the plot.
While most of the obtained communities are informative, a few communities were found to be more dispersed, approaching more than one topic. This might occur given the limitations of the embeddings model, since some words might not be available in the considered model. Despite these limitations, we show that the flow of information (from sentence to sentence) in the obtained semantic communities can be used to characterize texts.

Markov Chains
In order to capture how authors move from community to community (semantic field) as their story unfolds, we create a representation of community transitions. The idea of studying language via Markov process is not recent. One of the first uses of this model is the study of letters sequences [43]. Since then, Markov chains are used as a statistical tool in several natural language processing problems, including language modeling, machine translation and speech recognition [44].
Here we represented the transitions between semantic fields (network communities) as a first order Markov chain. In this representation, each community becomes a state. Note that this approach of representing communities as a single unit has also been used in other contexts [9]. The probabilities of transition are considered according to the frequency of transitions observed in adjacent sentences. As we shall show, using this model, it is possible was obtained with the method described in [9].
to detect patterns (see Section 3.5) of how authors change topics in their stories. As a proof of principle, these patterns are used to characterize texts in distinct classification tasks.
The process of creating a Markov chain from a network divided into communities is shown in Figure 4. In the previous phase, communities are identified to represent distinct semantic field of the story (see left graph in Figure 4). Because each sentence belongs to just one community, the text can be regarded as discrete time series, where each element corresponds to the membership (community label) of each sentence. Using this sequence of community labels, it is possible to create a Markov chain representing all transitions between communities (see graph on the top left of Figure 4). Transitions weights are proportional to the frequency in which they occur and normalized so as to represent a probability. This representation is akin to a Markov chain used in other works addressing the language modeling problem [45]. The main difference here is that we are not interested in the use of particular words, but in semantic fields [46]. Once the Markov chain is obtained, we characterize this structure using network motifs (see Section 3.5).

Motifs
Network motifs are used to analyze a wide range of complex systems, including in biological, social and information networks [47]. Motifs can be defined as small subgraphs occurring in real systems in a significant way. To quantify the significance, in general, one assumes an equivalent random network as null model. In text analysis, motifs have been used to analyze word networks [48] in applications focusing on the syntax and style of texts.
More recently, an approach based on labelled motifs showed that authors tend to use words in combination with particular motifs [49].
While the structure of the Markov Chains could be analyzed using traditional network measurements, we decided not to use these measurements owing to the limited size of these structures. As suggested in related works, a characterization based on network metrics in small networks might not be informative [50,14,3]. As we shall show in the results, this is a simple, yet useful approach to classify small Markov Chains.
Three different approaches were considered to extract motifs from Markov networks. We discriminated strategies according to the use of weights to count motifs (unweighted vs weighted). If a thresholding is applied before extracting motifs, the strategy is named with the "simplified": 1. Unweighted strategy: no thresholding is applied. All weights are disregarded. Every time a motif is detected, its frequency is increased by one.
2. Simplified unweighted strategy: this approach is the same as the unweighted strategy.
However, before counting motifs, the weakest edges are removed according to a given threshold.
3. Simplified weighted strategy: before counting motifs, the weakest edges are removed according to a given threshold. All edges weights are considered in the remaining Markov network. Every time a given motif is found, the respective "frequency" of that motif is increased by the sum of the weights of its edges.

Classification -Machine Learning Methods
The extracted motifs from the Markov Chains are used as input (features) to the classification systems. The following methods were used in the experiments: Decision Tree (CART), kNN, SVM (linear) and Naive Bayes [51]. The evaluation was performed using a 10-fold cross-validation approach. As suggested in related works, all classifiers were trained with their default configuration of parameters [52].

Classification tasks
Here we probed whether the dynamics of changes in semantic groups in books can be used to characterize stories. The proposed methodology was applied in two distinct classification tasks. In the first task, we aimed at distinguishing three different thematic classes: (i) In the first experiment, we evaluated if patterns of semantic changes are able to distinguish between children, philosophy or investigative books. We considered problems with two or three classes. The obtained results are shown in Table 1. In this case, weights were disregarded after the construction of the Markov networks (unweighted version). Considering subtasks encompassing only two classes, only the distinction between children and investigative texts were not significant, with a low accuracy rate. The distinction philosophy books and the other two classes, however, yielded a much better discrimination. These results were found to be significant. When all three classes are discriminated, a low accuracy rate was found (43.8%), even though this still represents a significant result. The low accuracy rate found using the proposed approach is a consequence of a regular behavior found in the Markov chains. In other words, in most of the books, all communities were found to be connected to each other, hampering thus the discriminability of different types of books. Given the low accuracy rates obtained with the unweighted strategy, we analyzed if the simplified unweighted version was able to provide a better characterization. In this case, the weakest edges were removed before the extraction of motifs. We considered the thresholding ranging between 0.01 and 0.20. The main idea here is to remove less important links between communities. The obtained results are shown in Table 2. All obtained results turned out to be significant. All previous accuracy rates were improved. Interestingly, a high discrimination rate (91.6%) was obtained when discriminating investigative and philosophy books. These results suggest that the threshold is an important pre-processing step here, given that it can boost the performance of the classification by a large margin. When combining thresholding and edges weights in the simplified weighted version, the results obtained in Table 3 were further improved. The highest gain in performance was observed when discriminating children from philosophy books: the performance improved from 81.0% to 89.0%. Only a minor improvement was observed when all three classes were discriminated. Overall, this results suggest that both thresholding and the use of edges weights might be useful to characterize Markov networks. Most importantly, all three methods showed that, in fact, there is a correlation between the thematic approached and the way in which authors approaches semantic groups in texts. We also investigated if the patterns of semantic flow varies with the publication date.
For this reason, we selected a dataset with books in different periods. The following classes were considered, according to the range of publication dates:  Table 3). Therefore, patterns of semantic flow seems to be less affected by the year of publication, while being more sensitive to the subject/topic approached by the text.

Conclusion
In this paper we investigate whether patterns of semantic flow arises for different classes of texts. To represent the relationship between ideas in texts, we used a sentence network representation, where sentences (nodes) are connected based on their semantic similarity.
Semantic clusters were identified via community detection and high-level representation of each book was created based on the transition between communities as the story unfolds.
Finally, motifs were extracted to characterize the patterns of transition between semantic groups (communities). When applied in two distinct tasks, interesting results were found. In the task aiming at classifying books according to the approached themes, we found an high accuracy rate (92.5%) when discriminating investigative and philosophy books. A significant performance in the classification was also obtained when discriminating books published in distinct epochs. However, the discriminability for this task was not as high as the ones obtained when discriminating investigative, philosophy and children books.
Given the complexity of the components in the proposed framework, we decided not to optimize each step of the process. Even without a rigorous optimization process, we were able to identify semantic flow patterns that were able to discriminate distinct classes of texts.
As future works, we intend to perform a systematic analysis on how to optimize the process.
For example, during the construction of the networks, different approaches could be used to link similar sentences. In a similar fashion, different strategies to identify communities could also be used in the analysis. Finally, we could also investigate additional approaches to characterize the obtained Markov networks.
The results obtained here suggests that different classes of books display different patterns of semantic flow. This suggests that the semantic flow could play an important role in other NLP tasks. For example, in the authorship recognition task, patterns extracted from a semantic flow analysis could be combined with other techniques to improve the characterization of authors. A similar idea could also be applied to the analysis of other stylometric tasks. Since semantic networks have been studied in cognitive sciences, we believe that the adopted network representation could be adapted and used -as an auxiliary tool -to study complex brain and cognitive processes that could assist the diagnosis of cognitive disorders via text analysis [53,54].