{'fraction_non_alphanumeric': 0.04450725615929801, 'fraction_numerical': 0.012107661154235573, 'mean_word_length': 4.654818702290076, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Mathematical Model for Linguistic Universals

Inspired by chemical kinetics and neurobiology, we propose a mathematical theory for pattern recurrence in text documents, applicable to a wide variety of languages. We present a Markov model at the discourse level for Steven Pinker’s “mentalese”, or chains of mental states that transcend the spoken/written forms. Such (potentially) universal temporal structures of textual patterns lead us to a language-independent semantic representation, or a translationally-invariant word embedding, thereby forming the common ground for both comprehensibility within a given language and translatability between different languages. Applying our model to documents of moderate lengths, without relying on external knowledge bases, we reconcile Noam Chomsky’s “poverty of stimulus” paradox with statistical learning of natural languages.

We human beings distinguish ourselves from other animals (1)(2)(3), in that our brain development (4-6) enables us to convey sophisticated ideas and to share individual experiences, via languages (7)(8)(9). Texts written in natural languages constitute a major medium that perpetuates our civilizations (10), as a cumulative body of knowledge. The quantitative mechanism underlying the mental faculties of language has long been a difficult problem for anthropologists, linguists, neurobiologists and psychologists (11)(12)(13)(14)(15), before attracting the attention of computer and data scientists (16)(17)(18)(19)(20)(21), in the recent wave of artificial intelligence. Instead of marveling at the partial success of data-hungry approaches (18)(19)(20)(21) to machine learning, we still crave for a cost-effective, interpretable and universal algorithm for understanding natural languagesone that mimics language acquisition and knowledge accumulation during early childhood, based on limited resources, as in Chomsky's "poverty of stimulus" scenario (11,15). Without filling the gap of data sizes, one cannot satisfactorily answer nativists' criticism (22) against empiricists' statistical models for natural languages.
Rising to the challenges outlined above, we perform a detailed mathematical analysis for computable "linguistic universals"-statistical patterns common to a wide range of human languages. On the theoretical side, we will present a stochastic "mentalese" model that depicts the timecourse of Markov states behind individual concepts. On the practical side, we will demonstrate (through automated word translation and question answering) that word's meaning can be numerically characterized by moderate-sized Markov neural networks, even when there is relatively scant data input.
Our Markov model explains, up to acceptably small error margins, how our innate language faculties (nature) may help us understand the world, by connecting dots of our past expe-riences (nurture), irrespective of our mother tongue. Bridging nature to nurture, our stochastic algorithm for Markov neural semantics reconciles the views of nativists and empiricists.
Heuristic background Languages differ in their phonemic repertoires ("elementary particles" in Jakobson's (24) terms), word morphologies ("atoms") and syntactic structures ("molecules"), corresponding to the three short time scales (phonological processing level, lexical level, and sentence level) in the Friederici hierarchy (13), which are mapped to different brain regions in functional magnetic resonance imaging (fMRI). These three Friederici scales exhibit no universal linguistic patterns and bear no semantic significance. Ferdinand de Saussure's foundational work (9) rules out semantic dependence on phonological representation (except for a limited set of onomatopoeias), while the inherent meaning of a word is affected by neither its morphological parameters (say, singular vs. plural, present vs. past) nor its syntactic rôles (say, subject vs. object, active vs. passive).
Based on the foregoing arguments, one might speculate that universal semantic content, or Pinker's "mentalese" (12), may only exist at the discourse level ("bulk materials", if we extrapolate Jakobson's (24) metaphor), namely, on the longest time scale in Friederici's neurobiological hierarchy (13). In this work, we turn such a qualitative speculation into a quantitative model (25). Concretely speaking, we observe the following statistical features of textual patterns (clusters of words that are morphologically related, see Fig. 1 and Fig. 3B for examples) shared by many languages in common: 1. The recurrence behavior of most textual patterns is consistent with time series generated by a certain Markov process, on the longest, as opposed to the shortest (26) Figure 1: Counting effective transitions between textual patterns. A transition from W i to W j is considered effective, if the underlined text fragment in between contains no occurrences of W i , and lasts longer than the longest word in W i ∪ W j . The reduced fragment length L i j (measured in the number of letters, punctuation marks and white spaces) discounts the length of the longest word in W i ∪ W j . We count waiting times in L i j , so as to ignore kinetic features (23) on the short time scales in the Friederici hierarchy, which may vary from language to language.
A B  fig. S3 for more versions.) The prominence of nearly real eigenvalues [satisfying arg λ(P) ≈ 0] is not found in a Markov matrix with random entries, whose spectrum is uniformly distributed over a circular disk centered at the origin (27, 28).

2.
Recurrence kinetics of a given concept nearly remains independent of the language in which it is expressed; 3. Kinetic data quantify the semantic distance between different textual patterns, thus allowing us to construct semantic fields by statistical computations.
These long-range temporal features of documents written in various languages, in our opinion, point to a universal kinetic mechanism that defines the semantic rôles of individual nodes in a web of words, mathematically and linguistically.
Transition probability, spectral invariance, pattern recurrence To begin, we show how to numerically construct a Markov matrix from a realistic text document, and how this Markov model enables us to interpret long-range temporal structures that are common to a wide variety of languages.
In our kinetic language model, we assume that (the gists of) texts are generated by a discrete Markov process on a semantic web with N nodes, each of which represents a textual pattern W k -a set of morphologically related content words (29), indexed by an integer k ∈ {1, 2, . . ., N}-occurring in a given document (30). The stochastic hoppings between the nodes are governed by an ergodic Markov transition matrix P = (p i j ) 1≤i, j≤N , which (putatively) caricatures the dynamics of mental activities (31) underlying a text, on the time scale of discourse level. We emphasize that our Markov model for the long-range behavior of human languages is independent of Chomsky's transformational generative grammar (32), the latter of which characterizes short-range syntactic features as hierarchical trees without Markovian structures.
One can estimate transition probabilities between textual patterns on short time scales, by simply counting unigrams and bigrams in a large corpus (33). To estimate long-range transition probabilities from documents of moderate lengths (e.g. a literary piece, a Wikipedia page), i.e. to learn despite a "poverty of the stimulus", we need some makeshift strategies.
Given a timecourse of molecular states in a biochemical reaction, we can partially reconstruct kinetic information (34) from the probability distribution for the waiting time between consecutive encounters of the same molecular state. Carrying this waiting time analysis a little further, we put a crude estimate of the transition probability p i j aš where n i j counts the number of effective transitions from W i to W j , and L i j is a statistic that measures the reduced fragment lengths of such transitions (Fig. 1). On the diagonal, the Gibbs weights n ii e − log L ii hearken back to the TF-IDF measure of word importance (35,36). Off the diagonal, the ensemble average log L i j weighs the cost of biochemical activation energy required to jump from W i to W j , so that the memorability factor e − log L i j can be viewed as a naïve estimate for the rate of associative learning per copy number n i j , in Hebb's fire-and-wire process (37). It is worth noting that our estimate ofp i j was based on statistical analysis of the text in situ, without digesting a document (or small parts of it) as a scrambled bag of words, a procedure implemented in conventional algorithms (16,18,20). The empirical Markov matrixP = (p i j ) 1≤i, j≤N has some desirable properties.
First, the components of the dominant eigenvectorπ (sat-isfyingπP =π) are nearly proportional to the word counts contributed by individual text patterns ( Fig. 2A). This is consistent with the fact that the ergodic Markov matrix P has a unique equilibrium state π (satisfying the invariant measure equation πP = π), whose vector components represent the probabilities of encountering individual Markov states.
Second, the spectrum σ(P) is approximately invariant against translations of texts (Fig. 2B). One can explain this

ROOMS REALLY LOVED HOUSE TALKI E N D G WAY PLEASI E
ji from the initial value r 1 ≈ 0.07, for matrix powersP n = (p (n) i j ) 1≤i, j≤100 constructed from four versions of Pride and Prejudice. (In contrast, one has r 1 ≈ 0.33 for a random 100 × 100 Markov matrix.) Such quick relaxations support our working hypothesis about detailed balance π i p i j = π j p ji . (B) Some textual patterns W i sorted by descending n ii , with font size proportional to the square root of the memorability factor e − log L ii (see scheme S2 for word stacking methods, and figs. S7-S20 for further examples). Topical (i.e. significantly non-Poissonian) patterns painted in red (resp. green) reside below (resp. above) the critical line of Poissonian banality (blue line in C), where the deviations exceed the error margin prescribed in (5). (C) Recurrence statistics for textual patterns (gray, red and green dots with radii 1 ). Labels for proper names and some literary motifs are attached next to the corresponding colored dots. By Jensen's inequality, all the data points must sit below the green dashed line with unit slope and zero intercept. Actually, almost all data points lie beneath [up to error margin prescribed in (5) through a thought experiment involving two monolingual subjects, Alice and Bob, whose native languages are A and B, respectively. Suppose that we ask Alice to measure the transition probability from words in language A to words in language B. Alice first processes the input in her native language by a Markov matrix P A , and then translates into language B, using a dictionary matrix T A→B . When Bob is assigned the same task, he needs to first translate the input into language B, using the same dictionary T A→B , before brainstorming in his own native language, using P B . Putatively, semantic content is universal (shared by native speakers of different languages), so we must have In the ideal scenario where translation is lossless (with invertible T A→B ), the Markov matrices P A and P B are indeed linked to each other by a similarity transformation that leaves their spectrum intact. Third, the empirical matrixP approaches detailed balance after a few iterations (Fig. 3A). Since textual patterns typi-cally recur on the longest time scale in the Friederici hierarchy, we may further assume, for the sake of convenience, that the ergodic Markov transition matrix P honors the detailed balance condition π i p i j = π j p ji for 1 ≤ i, j ≤ N.
On a Markov chain with detailed balance, the recurrence time τ = L ii of any individual state W i is distributed as a weighted superposition of exponential decays (see fig. S1 for multi-exponential fits) where c m , k m > 0, and ∑ m c m = 1 , a functional form that frequently crops up in dynamic studies of biological macromolecules (34). The multi-exponential decay laws impose an inequality constraint: where γ 0 = 0.57721566... is the Euler-Mascheroni constant.   We postulate that a banal textual pattern tends to appear in a document in a memoryless manner, as in a Poisson process ( fig. S1B). Therefore, the effective time elapse τ = L ii between consecutive encounters of a banal W i obeys an exponential distribution P(L ii > t) ∼ e −kt for a single kinetic rate constant k > 0. The equality log L ii − log L ii + γ 0 = 0 (blue line in Fig. 3C) marks Poissonian banality when the text is infinitely long. If we have n ii independent samples of exponentially distributed random variables L ii , then the statistic δ i := log L ii − log L ii − γ 0 + 1 2n ii satisfies an inequality with probability 95%. As a working definition, we consider a textual pattern topical, if its δ i value violates the aforementioned inequality (Figs. 3B and C), corresponding to significantly non-Poissonian recurrence statistics. If a textual pattern W i qualifies as a topic by our definition, then the signals in its coarse-grained timecourse [say, a vector b i = (b i,1 , . . . , b i between two vectors with nonnegative entries, allow us to align some topics found in parallel versions of the same document, in languages A and B (Fig. 3D). (See fig. S12B for a control experiment on nontopics.)

Local graphs, kinetic semantics, machine translation
We have just shown that diagonal kinetic data (n ii and L ii statistics) can be used to extract topics numerically, and a small fraction of such topics have vector representations on the scale of chapters, which are stable against translations. Our next goal is to computationally construct semantic fields from off-diagonal kinetic data (n i j and L i j statistics), and vectorize topics without coarse-graining the text document, in a language-independent manner, valid on all the temporal scales above the discourse level.
To achieve this, we first need to specify a local, directed, and weighted graph (corresponding to a localized Markov transition matrixP [i] ) for each topical pattern W i . To localize, we remove edges between two vertices W i and W j , when their average temporal distance log L i j exceeds a statistical threshold to be defined below.
If we have a Markov process on a semantic web that honors detailed balance, then for each fixed textual pattern W i , we can determine the distribution of hitting times L i j , 1 ≤ j ≤ N from that of the return times L ii (40). Further assuming that the distribution of log L i j , 1 ≤ j ≤ N is nearly Gaussian, we can use the following approximation: where Here, the estimate "≈" in (6) becomes asymptotically exact if distinct textual patterns are statistically independent (such as α 13 , α 24 , α 31 , α 34 in Fig. 4A).  Figure 5: Semantic similarities s(W en i , W fr j ) between selected topics (sorted in descending n ii ≥ 20) in two versions of Pride and Prejudice. Rows and columns filled with zeros are not shown. Cross-hairs meet at optimal nodes that solve the bipartite matching problem (see figs. S8-S20 for more examples). The thickness of each horizontal (resp. vertical) cross-hair is inversely proportional to the rowwise (resp. column-wise) ranking of the similarity score for the optimal node. Green (resp. amber) cross-hair indicates an exact (resp. a close but non-exact) match. At the same confidence level (0.7) for similarities, this experiment has better recall than Fig. 3D, without much cost of precision.
Empirically, we find that higher α i j (ℓ) scores point to closer affinities between textual patterns (Fig. 4A), attributable to kinship (Elizabeth, Jane), courtship (Darcy, Elizabeth), disposition (Darcy, pride) and so on. Therefore, even in a "poverty of stimulus" scenario, without references other than the novel itself, we can use the α i j (ℓ) scores as guides to numerical approximations of semantic fields, hereafter referred to as semantic cliques.
We invite a topical pattern W j to the semantic clique S i (insets of Figs. 4A and B) surrounding . that is associated with the standard Gaussian distribution. On a local graph with vertices S i = {W i 1 = W i , W i 2 , . . . , W i N i }, we specify the connectivity of each directed edge by a localized Markov ma- jk ) 1≤ j,k≤N i , which is row-renormalized from an N i × N i subblock ofP with the same set of vertices as S i . Resetting the entriesp [i] 1k andp [i] j1 as zero, one arrives at the localized recurrence matrixŘ [i] , which characterizes the distribution for recurrence times to the Markov state W i in S i . Experimentally, we resolve the connectivity of an individual pattern W i through the recurrence spectrum σ(Ř [i] ) (Fig. 4B). The dominant eigenvalues ofŘ [i] are concept-specific while remaining nearly language-independent. Such empirical evidence motivates us to quantify the kinetic profile of a textual pattern W i by a descending list for the magnitudes of eigenvalues v i = (|λ 1 (Ř [i] )|, |λ 2 (Ř [i] )|, . . .). We zero-pad this vector from the (⌊e η i ⌋ + 1)st component onwards, where jk is the entropy production rate (41) of the Markov matrixP [i] , measured in nats per word.
Given a textual pattern W A i in language A, the aforementioned vector embeddings allow us to numerically locate a semantically close pattern in a parallel text (42) written in another language B, in two steps: (1) Divide the document into K chapters, and define the semantic similarity function as s( (2) Solve a bipartite matching problem (Fig. 5)

⇒
(1) Three-phase electric power is a common method of alternatingcurrent electric power generation, transmission, and distribution.
(2) Three-phase transformer with four wire output for 208Y/120 volt service: one wire for neutral, others for A, B and C phases (3) A three-phase system is usually more economical than an equivalent single-phase or two-phase system at the same voltage because it uses less conductor material to transmit electrical power.
(4) Three-phase electric power transmission (5) It is also used to power large motors and other heavy loads.  Figure 6: a A construction of semantic clique Q ∪ Q ′ weighted by π and subsequent query expansion. Top 5 candidate answers shown with font sizes proportional to the entropy production score in (8 • Synonyms (pride and vanity in English, orgeuil and fierté in French, etc.); • Temperaments (Elizabeth, a delightful girl, often laughs, corresponding to French verbs sourire and rire); • Co-references (e.g. Darcy as a personification of pride); • Causalities (such as pride based on fortune), as well as other kinds of relations typically shared by words within a semantic field. Now, we can exploit the connectivity of these semantic cliques S i to build a rudimentary question-answering machine.
Concretely speaking, we are given a document (of moderate length) and a natural language question as input, and want to rate and rank the sentences within the document by their relevance to the question, containing topical patterns Q = {W q 1 , . . . , W q K }. We expand the query into Q ∪ Q ′ , a union of semantic cliques: Q ∪ Q ′ = K k=1 S q k . As before, we can construct a localized Markov matrix P = (p i j ) 1≤i, j≤N on this subset of textual patterns Q ∪ Q ′ . We further use the Brin-Page damping (46) to derive an ergodic Markov matrix P = ( p i j ) 1≤i, j≤N , where p i j = 0.85p i j + 0.15 N . By analogy to the behavior of internet surfing (46,47), we model the process of associative reasoning (48) as a navigation through the nodes Q ∪ Q ′ according to P, which quantifies the click-through rate from one idea to another. The PageRank recursion (47) ensures a unique equilibrium state π attached to P. If our question Q and a candidate answer A contain, respectively, words from textual patterns W Q 1 , . . . , W Q m ∈ Q and W A 1 , . . . , W A n ∈ Q ∪ Q ′ (counting multiplicities, but excluding function words and patterns with fewer than 3 occurrences in the reference document), then we assign the following entropy production score to this question-answer pair. A sample work flow is shown in Fig. 6A, to illustrate how our machine answers a question, using a Wikipedia page (without infoboxes and other structural data) as its only reference document and training source. It can be seen that our numerical associative reasoning generates a weighted set of nodes Q ∪ Q ′ (presented graphically as a thought bubble in Fig. 6A) that is comparable to the output from a typical human reader of Wikipedia, without the help of external stimuli or knowledge feed. We test our model (EZ in Fig. 6B) on all the 1242 questions in the WikiQA data set, each of which is accompanied by at least one correct answer located in a designated Wikipedia page. Our algorithm's performance is roughly on par with LCLR and CNN benchmarks (49), improving upon the baseline by significant margin. This is perhaps remarkable, considering the relatively scant data at our disposal: unlike the LCLR approach, our numerical discovery of synonyms does not draw on the WordNet database or pre-existent corpora of question-answer pairs; unlike the CNN method, we do not need pre-trained word2vec embeddings (18) as semantic input.
Moreover, our algorithm (EZ * in Fig. 6B) performs slightly better on a subset of 990 questions that do not require quantitative cues (How large? How long? How many? How old? What became of? What happened to? What year? and so on). This indicates that our structural model fits associative reasoning better than rule-based reasoning (48), while imitating human behavior in the presence of limited data.
Discussion and outlook In our current work, we develop a stochastic model that assigns meaning (semantic content) to words, via association and connectivity. Consistently using a single Markov framework, we are able to recover word frequencies ( Fig. 2A), extract topics (Fig. 3B), interpret recurrence statistics (Figs. 3A and C), and ultimately demonstrate translatability (Figs. 2B, 3D, 4B, 5) as well as comprehensibility (Figs. 4A, 6A) through statistical learning of languages. Hopefully, our Markov model at the discourse level offers an approximate explanation for the universal hardware part in our language faculties, which are mechanisms common to all human minds. It is not our purpose to efface the diversity of the software aspects of human languages: the phonological encoding scheme adapts to geographical and climate conditions of native speakers (50)(51)(52), while lexical and syntactical features are highly susceptible to social, cognitive and cultural factors (53)(54)(55)(56).
Being a caricature of the universal human nature (57) on the longest neuro-linguistic time scale (universal up to matrix similarity transformations like T −1 A→B P A T A→B = P B ), our mathematical theory for textual patterns functions even in the "poverty of stimulus" regime, with modest computa-tional complexity. On the practical side, our Markov model achieves some reasonable results in word translation and text comprehension. This suggests that we are probably close to a complete set of semantic invariants, after demystifying the long-range behavior of human languages. In the near future, we hope to extend our framework further, to incorporate both Markovian and non-Markovian structures across all the four Friederici neurobiological hierarchies-The Mathematical Principles of Natural Languages, as we envision, must and will combine the statistical analysis of a Markov model with neurolinguistic properties on shorter time scales (22).