{'fraction_non_alphanumeric': 0.03554135192717412, 'fraction_numerical': 0.00537478210342824, 'mean_word_length': 4.091962524654832, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Regular decomposition of large graphs and other structures: scalability and robustness towards missing data

A method for compression of large graphs and matrices to a block structure is further developed. Szemer\'edi's regularity lemma is used as a generic motivation of the significance of stochastic block models. Another ingredient of the method is Rissanen's minimum description length principle (MDL). We continue our previous work on the subject, considering cases of missing data and scaling of algorithms to extremely large size of graphs. In this way it would be possible to find out a large scale structure of a huge graphs of certain type using only a tiny part of graph information and obtaining a compact representation of such graphs useful in computations and visualization.


I. INTRODUCTION
So called 'Big Data' is a hot topic in science and applications. Revealing and understanding various relations embedded in such large data sets is of special interest. One good example is the case of semantic relations between words in large corpora of natural language data. Relations can also exist between various data sets, forming large high-order tensors, thus requiring integration of various data sets. In such a scenario, algorithms based on stochastic block models (SBM, also known as generalized random graph) [1] (for a review see e.g. [2]) are very attractive solutions instead of simple clustering like k-means.
It is also natural to look what the abstract mathematics can offer. A strong result, less known among practicing specialists, is so called Szemerédi's Regularity Lemma (SRL), [3] that in way supports SBM-based approach to large data analysis. SRL proves the existence of a SBM-like structure for any large graph and similar objects like hyper-graphs. SRL has been a rich source in pure mathematics -it appears as an element of important results in many fields. We think that in more practical research, like ours, it is good to be aware of the broad context of SRL as a guiding principle and a source of inspiration.
The methodology of Regular Decomposition (RD) has been developed in our previous works [5], [6], [7], [8]. In [8] we analyzed RD using information theory and Rissanen's Minimum Description Length Principle (MDL) [9]. Our theoretical description is partly overlapping with SBM literature using the apparatus of statistical physics, e.g., extensive works by T.P. Peixoto. In particular, the use of MDL to find the number of blocks has been studied earlier by Peixoto [11].
We focus on the case of very large and dense simple graphs. Other cases like rectangular matrices with real entries can be treated in very similar manner using the approach described in [8]. We wish to demonstrate that large scale block structures of such graphs can be learned from a bounded size, usually quite small, samples. The block structure found from a sample can then be extended to the rest of the graph in just a linear time w.r.t. the number of nodes. This means that a largescale structure of graphs can be found from a limited sample without extensive use of adjacency relations. Only a linear number of links is needed, although the graph is dense with a quadratic number of links. Such a structure is probably enough for many applications like those involving counting of small subgraphs. It also helps in finding a comprehensive overall understanding of graphs when the graph size is far too large for complete mapping and visualization. The graph data can also be distributed, and the process of labeling can then be distributed extremely efficiently since the classifier is a simple object. We also introduce a new version of Regular Decomposition for simple graphs that tolerates missing data. Such an algorithm helps its use in the often encountered situation that part of the link data is missing.
Our future work will be dedicated to the case of sparse graphs, which is the most important in big data. Here we could merge well-clustered local subgraphs into super-nodes and study graphs of such super-nodes. Provided that we obtain dense graphs, regular decomposition could be used efficiently. SRL has a formulation for the sparse case that can be used as a starting point, along with other ideas from graph theory and practices, to create RD algorithms that are efficient with sparse data.

II. REGULAR DECOMPOSITION OF GRAPHS AND MATRICES
SRL states, roughly speaking, that the nodes of any large enough graph can be partitioned into a bounded number, k, of equally sized 'clusters', and one small exception set, in such a way that most pairs of clusters look like random bipartite graphs with independent links, with link probability equal to the link density. SRL is most significant for large and dense graphs. However, a similar result is extended also to many other cases and structures, see the Refs. in [4] with a constant flow of significant new results.
In RD, we simply replace random-like bipartite graphs by a set of truly random bipartite graphs and use it as a modeling space in the MDL theory. The next step is to find an optimal model that explains the graph in a most economic way using MDL [8]. In the case of a matrix with non-negative entries, we replace random graph models with a kind of bipartite Poissonian block models: a matrix element a i,j between row i and column j is thought to correspond to a random multi-link between nodes i and j and the number of links is distributed as a Poisson random variable with mean a i,j . The bipartition is formed by the sets of rows and columns, respectively. Then the resulting RD is very similar to the case of binary graphs. This approach allows also the analysis of several inter-related datasets, for instance, using corresponding tensor models. RD also promises extreme scalability as well as tolerance against noise and missing data.
In RD, the code for a simple graph G = (V, E) with respect to a partition ξ = {A 1 , . . . , A k } of V has a length at most (and, for large graphs, typically close to) where l * (m) := log(m)+log log(m)+· · · is the coding length of an integer m and is the entropy of the Bernoulli(p) distribution. A corresponding representation exists also for the coding length of a matrix, interpreted as an average of a Poissonian block model, see details in [8]. Figure 1 shows is a typical result with regular decomposition of a symmetric real matrix. The regular groups are seen as almost flat rectangular blocks that are revealed by a permutation of rows and columns of the left hand side matrix. For a non-symmetric rectangular matrix, the visual effect of RD is similar: a "chessboard" that is irregular in the sizes and shapes of the rectangles.

A. Matrix formulation of RD algorithm for graphs
Next we present algorithms that we have used in actual computations of regular decompositions of graph and matrix data [8]. These are written corresponding to (1) (except that L 1 (G|ξ) is neglected, being insignificant even in the rigorous analysis [8]). The code lengths L 4 and L 5 have a usual interpretation as the minus log-likelihood of a graph corresponding to a stochastic block model. The version with missing data, presented at the end, is a new contribution. These algorithms use matrix algebra and are supposed to work with moderate size data. In the next section, we propose a sampling-based algorithm that allows the scaling of RD for big data.
A partition of a set of n nodes into k nonempty sets, called the blocks, can be described by an n × k binary matrix such that all row sums equal to one and none of the column sums equals zero. The latter condition indicates that the sets of the partition are non-empty. Denote the space of all such matrices by R k and the members of this set as R ∈ R k . Definition 2.1: For a given graph G with n nodes, adjacency matrix A and a partition matrix R ∈ R k , denote where · T stands for matrix transpose. Block sizes are the column sums of R and are denoted as The number of links within each block and between block pairs are denoted and computed as Finally, define k × k link-density matrix Then the length of the code that uniquely describes the graph corresponding to A, using the model (R, P ), appears to be: Definition 2.2: presents the code length of the model. The two-part MDL program of finding the optimal model, denoted as R k * , can now be written as: To solve this program approximately, we can use the following greedy algorithm.
n}, such that the two-part code for G is close to the shortest possible for all possible block models with number of blocks in the range from 1 to n.
and STOP the program. 6. GoTo 1.
Definition 2.4: A mapping Φ : R k → R k is defined as follows. First, define following matrices element-wisely: where we set by definition, log 0 := 0 since it appears in combination 0 log 0 = 0 in matrix L, and α, β ∈ {1, 2, · · · , k}, and and finally define the matrix function Φ(·) element-wisely on argument R: The mapping Φ(R) defines a sparse optimization of partition R, where each node is re-placed to new block independently of all other placements; hence the term 'greedy algorithm'. Algorithm 2.5: ARGMAX k Algorithm for finding regular decomposition for fixed k. Input: A: the adjacency matrix of a graph (an n×n symmetric binary matrix with zero trace); N : an integer (the number of iterations in the search of a global optimum); k: a positive integer. Start: m = 1. 1. i := 0; generate a uniformly random element R i ∈ R k . 2. If at least one of the column sums of R i is zero, GoTo 1, if not, then compute: In ARGMAX k the outer loop 1-5 runs several (N ) optimization rounds finding each time a local optima, finally the best local optima is the output. At each optimization round, the inner loop 2-3, improves the initial random partition until a fixed point is reached and no further improvements of the partition are possible. This fixed point is an indication that a local optimum is found.
For very large graphs, the program may not be solvable in the sense that it is not possible and reasonable to go through all possible values of k ∈ {1, 2, · · · , n}. One option is to limit the range of k. In case that no minimum is found, then use as an optimal choice the model found for the largest k within this range. Another option is to find the first minimum with smallest k and stop. When the graph is extremely large, it makes sense to use only a randomly sampled sub-graph as an input -indeed, when k * << n, a large-scale structure can be estimated from a sample, as described in more details in the next section.

B. Regular decomposition with missing link information
Now assume that part of the link information is lost. We only consider the simplest case when the link information (whether there is a link or not) is lost uniformly over all node pairs. We also exclude possibility of a sparse representation of graphs, when only those node pairs that have links are listed, since this would lead to unresolvable ambiguity in our setting since if some node pair is not in the list it could mean two things: there is no link or the link information is not available.
We just formulate how the matrices used in previous section are modified. The RD algorithms themselves are unaltered.
Denote by A the adjacency matrix with missing link values (0 or 1) replaced by −1's. A is symmetric and we also assume that we know its true dimension (the number of nodes). Put −1 on diagonal of A, just for convenience of further formulas.
where the absolute value of a matrix is defined element-wise. Thus, D has the same elements as A, except that the entries equal to −1 are replaced by 0. Next define a 'mask matrix' where +1 means element-wise addition of 1 to all elements of the matrix. That is, b i,j = 0 iff a i,j = −1, and b i,j = 1 otherwise. In other words, B is the indicator of known link data in A. The matrix P 1 is now: The number of observed links in a block indexed by (α, β) is We also need the following matrix, whose elements present the number of node pairs in block (α, β) that contain link data: The P -matrix is defined as the link densities in the observed data: (P (R)) α,β := 1 {n α,β >0} e α,β n α,β .
As stated above, we assume that the matrix elements of P be close to the corresponding actual link densities in the case that we would possess complete link data. This also assumes that the graph and the blocks must have large size, which is not a big constraint since RD is aimed at the analysis of large graphs. The cost function in Definition 2.2 remains unaltered, although now the P -matrix elements are only estimates based on observed data and not the exact one. The block sizes are true ones that are known since we assume that we know the actual number of nodes in the graph. The interpretation is that the contributions of missing data entries in a block are replaced by the average contribution found from the observed data of that block. It should be noted that similar scaling must be done in the RD algorithms that find the optimum of the cost function. E.g., A should be replaced with D, and summing over elements of D should be rescaled so that the contribution is proprtional to the true size of the corresponding block.

A. Sampling scheme and experimental testing
Assume that a very large graph is generated from a SBM. Denote the blocks or node clusters by V 1 , V 2 , · · · , V k and the link probability matrix by P , that is, a k ×k symmetric matrix with entries in (0, 1). We also assume that no two rows are identical, to avoid redundant structure. Then we generate links between blocks and within blocks using P as independent link probabilities. The size of the graph G is assumed large and denoted by N . Now take a uniform random sample of size n of the nodes. Denoted the graph induced by sampling as G n . The relative sizes of blocks are denoted as r i := |Vi| N . The probability of sampling a node from block i is r i . We want to test whether the block structure of G can be reconstructed from a small sample, the size of which does not depend on N , using a classifier with time complexity ∼ N . We believe that the answer is "yes".
For simplicity, assume that we know the block structure of G n by having run an ideally working RD-algorithm. This is not too far from reality, although some of the nodes may in reality be misclassified. However, there is not a big difference, if most of the nodes are correctly classified and n is not very small. The real graphs, however, are not generated from any SBMs, and this is a more serious drawback. In the latter case we would like to show that it is not necessary to run RD on G, rather it is sufficient to run RD on G n and just classify the rest of the nodes based on that structure. The linkage between the SBM case and the case of real data is not rigorous and needs further work.
In G n , we denote the blocks asV 1 ,V 2 , · · · ,V k , and the empirical link densities between these blocks asd i,j . The relative sizes of the blocks as well as the link densities deviate in G n from their counterparts in G. Now assume that we take a node i ∈ V β outside G n together with its links to G n . What is the probability that the RD classifier places this node intô V β ?
Denote by e α (v) the number of links from v toV α , n α := |V α |. The RD classifier based on G n is the following program: whereξ = V 1 ,V 2 , · · · ,V k andd stands for the k ×k matrix with matrix elementsd i,j . To compute the optimal block for node i we need to compute and check a list of k numbers as well as compute the densities e α (i). The computation time is upper-bounded by Θ(kn). For a size n that is independent on N , the computation takes only a constant time. By this we mean that it is enough to have some constant n regardless how large the graph is. Such an n depends on the relative sizes of blocks and on k. This is because it is obviously necessary to have enough samples from each block to be able to obtain good estimates of link densities and if some blocks are very small, more samples are needed.
We made experiments with k = 20 and with equal relative sizes of blocks. The P -matrix elements were i.i.d. Uniform(0, 1) r.v.'s. In this case, already n > 200 is such that no classification error was found in a long series of experiments with repetitions of random samples and classification instances. Of course, errors are always possible, but they become insignificant already when each block has about 20 members. A similar situation is met with non-equal block sizes, but then the small blocks dictate the required n. When the smallest block has more that a couple of dozen nodes, the errors become unnoticeable in experiments.
We conjecture that if such a block structure or close to it exists for a large graph, it can be found out using only a very small subgraph and a small number of links per node in order to place every node into right blocks with only very few errors. This would give a coarse view on the large graph useful for getting an overall idea of link distribution without actually retrieving any link information besides labeling of its nodes. For instance, if we have a large node set, then the obtained labeling of nodes and the revealed block densities would allow computation of, say, the number of triangles or other small subgraphs, or the link density inside a set. It would be possible to use this scheme in case of dynamical graph, to label new nodes and monitor the evolution of its large scale structure. We can also adapt our RD method to the case of multi-layer networks and tensors and etc. A similar sampling scheme would also be desirable and probably doable. On the other hand, large sparse networks need principally new solutions and development of a kind of sparse version of RD. These are major directions of our future work.

B. Mathematical framework for rigorous analysis
In our future work we aim at rigorous analysis of errors in the sampling scheme described in the previous section.
Here we give some more convenient formulations that we shall use in our future work. They also clarify the character of mathematical objects we are dealing with.
Obviously, we can generate a graph process G n (V n , E n , ξ n ) of increasing realizations of Blockmodel(r, P ) by starting with a single vertex and adding at every step n + 1 one new vertex v n+1 , its label κ(v n+1 ) and its edges to V n . Now assume that the labellings are hidden after some n 0 , and only the edges are observed. Instead, each label κ(v n ), , Fig. 7. A scheme of RD for a huge graph shown at the top, in reality we assume much larger graph than what is shown in the picture. First a moderate size sample of n 0 nodes and induced subgraph is formed. RD is applied to this small graph, groupsξn 0 and matrixP are found, shown as the graph in the middle. The sequentially any node v from the big graph can be classified to a regular group using just counts of links of this chosen node to regular groups of the sample graph. This classification requires a constant number of computations, upper bounded by k 2 n 0 , with elementary functions. As a result nodes of the whole graph can be classified in just linear time with respect to number of nodes. After the classification is done (shown in the ring shape structure at the right side) the RD of the large graph is done simply by retrieving link information. The result is shown in the lower graph.
n > n 0 , is estimated based on the edge set e(v n , V 0 ) and the empirical model Blockmodel(r,P ). In this way, a sequence of classificationsξ n , n > n 0 , is generated. How different are (ξ n ) and (ξ n )? Note that, given (V n0 , E n0 , ξ n0 ), the set-valued process {w ∈ V 0 : (v n , w) ∈ E n } is i.i.d., and therefore also the processκ(v n ) is i.i.d. for n > n 0 .
The maximum likelihood classifierκ(·) has the form κ(v) = arg min j C j (v|ξ n0 ,P , k). Denote Denote the Kullback-Leibler divergence from Bernoulli(p) to Bernoulli(q) as Adding and subtracting H(q i (v)) to each summand in (3), the expression to be minimized can be written as Define also the "ideal (V n0 , E n0 , ξ n0 ) based classifier" κ * (v) = arg min j k i=1 r i I(q i (v) : p ji ).
In future work we try to find proofs and conditions for the described sampling scheme assuming also partially lost data and without the assumption that graph is generated using a SBM. Here the 'Sampling lemma' 2.3 from [4] can be handy, proving that in general setting the link densities can be found from small samples. Even more intriguing is the question of using RD for sparse graphs and finding some kind of analog of RD-approach in this situation.