{'fraction_non_alphanumeric': 0.030307879408564296, 'fraction_numerical': 0.008449081482429751, 'mean_word_length': 4.862476547842402, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing

This article proposes the use of Vector Symbolic Architectures for implementing Hierarchical Graph Neuron, an architecture for memorizing patterns of generic sensor stimuli. The adoption of a Vector Symbolic representation ensures a one-layered design for the approach, while maintaining the previously reported properties and performance characteristics of Hierarchical Graph Neuron, and also improving the noise resistance of the architecture. The proposed architecture enables a linear (with respect to the number of stored entries) time search for an arbitrary sub-pattern.


I. INTRODUCTION
G RAPH Neuron (GN) is an approach for memorizing patterns of generic sensor stimuli for later template matching. It is based on the hypothesis that a better associative memory resource can be created by changing the emphasis from high speed sequential CPU processing to parallel network-centric processing [2], [3]. In contrast to contemporary machine learning approaches, GN allows introduction of new patterns in the learning set without the need for retraining. Whilst doing so, it exhibits a high level of scalability i.e. its performance and accuracy do not degrade as the number of stored patterns increases over time.
Vector Symbolic Architectures (VSA) [4] are a bio-inspired method of representing concepts and their meaning for modeling cognitive reasoning. It exhibits a set of unique properties which make it suitable for implementation of artificial general intelligence [5], [6], [7], and so, creation of complex systems for sensing and pattern recognition without reliance on complex computation. In the biological world, extremely successful applications of these approaches can be found. One example is the ordinary house fly. A house fly is capable of conducting very complex maneuvers albeit it possesses very little computational capacity 1 . Another interesting biological example is the compound eyes of arthropods. These compound eyes consist of large number of sensors with limited and localized processing capabilities for performing relatively complex sensing tasks [9].
This article presents contributions in two domains: organization of associative memory and properties of connectionist distributed representation. In the first area the article introduces a novel bio-inspired architecture, called Holographic Graph Neuron (HoloGN), for one-shot pattern learning, which is build upon Graph Neuron's flexible input encoding abstraction and strong reasoning capabilities of the VSA representation. In the second area this article extends understanding of the performance proprieties of distributed representation, which opens a way for new applications.
The article is structured as follows. Section II presents an overview of the work related to the matter presented in this article. The background information on the theories, concepts and approaches used in HoloGN is described in Section III. Sections IV through VI present the main contribution of this article -the design of the HoloGN architecture and its performance characteristics. The conclusions are presented in Section VII.

II. RELATED WORK
Associative memory (AM) is a sub-domain of artificial neural networks, which utilises the benefits of content-addressable memory (CAM) [10] in microcomputers. The AM concept was originally developed in an effort to utilise the power and speed of existing computer systems for solving large-scale and computationally intensive problems by simulating biological neurosystems.
The Hierarchical Graph Neuron (HGN) approach [2] is a type of associative memory which signifies the hierarchical structure in its implementation. Hierarchical structures in associative memory models are of interest as these have been shown to improve the rate of recall in pattern recognition applications. The distributed HGN scheme also allows for better control of the network resources. This scheme compares well with contemporary approaches such as Self-Organizing Map and Support Vector Machine in terms of speed and accuracy.
Vector Symbolic Architectures [5] are a class of connectionist models that use hyper-dimensional vectors (i.e. vectors of several thousand elements) to encode structured information as distributed or holographic representation. In this technique structured data is represented by performing basic arithmetic operations on field-value tuples. Distributed representations of data structures are an approach actively used in the area of cognitive computing for representing and reasoning upon semantically bound information [4], [11].
In [12] a VSA-based knowledge-representation architecture is proposed for learning arbitrarily complex, hierarchical, symbolic relationships (patterns) between sensors and actuators in robotics. Recently the theory of hyper-dimensional computing, and VSA in particular, were adopted for implementing novel communication protocols and architectures for collective communications in machine-to-machine communication scenarios [13], [14]. The first work demonstrates unique reliability and timing properties essential in the context of industrial machineto-machine communications. The latter work shows the feasibility of implementing collective communications using current radio technology. This article presents an algorithmic ground for further design of the distributed HoloGN on top of the architecture presented in [13].

III. OVERVIEW OF ESSENTIAL PARTS OF RELEVANT CONCEPTS AND THEORIES
A. Hierarchical Graph Neuron Figure 1 illustrates the Hierarchical Graph Neuron approach. Consider only the bottom layer of the construction without the hierarchy of upper nodes; this bottom level is the original flat network of Graph Neurons [2]. Each GN is a model for a set of generic sensory values. When seen as a network, graph neurons can be modeled by an array where columns are individual GNs and rows are possible symbols, which a neuron can recognize. For example, if there are only two possible symbols, say "X" and "Y", in the alphabet of a pattern, then only two rows are needed to represent those symbols. The number of columns 2 in the GN array determines the size of patterns, which it can analyse.
An input pattern is defined as a stimulus produced within the network. In Figure 1, each GN can analyse a symbol ("X" or "Y") of a pattern comprising of five elements. In each GN (column) only the element with matching value (a row ID) would respond. For example, if the pattern is "YXYYX", then in the second column "X" element will be activated as response to this input stimulus. If a particular element in the GN-column is activated it sends a report to all adjacent GNs. The report contains the activated GN's element ID (the row index). Otherwise, it simply ignores the stimulus and returns to the idle state.
During the next phase, all GNs communicate the indices of the activated elements with the adjacent columns at their level, and additionally communicate the stored bias information to the layer above. The procedure continues in an ascending manner until the collective bias information reaches the top of the hierarchy. The higher level GNs can thus provide a more authoritative assessment of the input pattern. The accuracy of HGN was demonstrated to be comparable to the accuracy of Neural Network with back-propagation [2].

B. Fundamentals of Vector Symbolic Architecture and Binary Spatter Codes
Vector Symbolic Architecture is an approach for encoding and operations on distributed representation of information. VSA has previously been used mainly in the area of cognitive computing for representing and reasoning upon semantically bound information [4], [15].
The fundamental difference between distributed and localist representations of data is as follows: in traditional (localist) computing architectures each bit and its position within a structure of bits are significant (for example, a field in a database has a predefined offset amongst other fields and a symbolic value has unique representation in ASCII codes); whereas in a distributed representation all entities are represented by binary random vectors of very high dimension. These representations are also called Binary Spatter Codes (BSC), though for the remainder of the article we use term HD-vector when referring to BSC codes, for reasons of brevity. High dimensionality refers to that fact that in HDvectors, several thousand positions (of binary numbers) are used for representing a single entity; [4] proposes the use of vectors of 10000 binary elements. Such entities have the following useful properties: 1) Randomness: Randomness means that the values on each position of an HD-vector are independent of each other, and "0" and "1" components are equally probable. In very high dimensions, the distances from any arbitrary chosen HD-vector to more than 99.99 % of all other vectors in the representation space are concentrated around 0.5 Hamming distance. Interested readers are referred to [4] and [16] for comprehensive analysis of probabilistic properties of the hyperdimensional representation space.
When d is in the range of several thousand binary elements, the calculation of the binomial coefficient requires sophisticated computations. The following equations use an approximation of the binomial distribution, via the de Moivre-Laplace theorem: 2) Similarity metric: A similarity between two binary representation is characterized by Hamming distance, which (for two vectors) measures the number of positions in which they differ: where a i , b i are bits on positions i in vectors A and B of dimension d, and where ⊗ denotes the bit-wise XOR operation.
3) Generation of HD-vectors: Random binary vectors with the above properties can be generated with Zadoff-Chu sequences [17], a method widely used in telecommunications to generate pseudo-orthogonal preambles. Using this approach a sequence of K vectors, which are pseudo-orthogonal to a given initial random HD-vector A (i.e. Hamming distance between them approximately equals 0.5) are obtained by cyclically shifting A by i positions, where 1 < i ≤ K ≤ d. Further in the article this operation is denoted as Sh(A, i). The cyclic shift operation has the following properties: Note that the cyclic shift is a special case of the permutation operation [4]. In the context of VSA, permutations were previously used to encode sequences of semantically bound elements.

4) Bundling of vectors:
Joining several entities into one structure is done by the bundling operation; it is implemented by a thresholded sum of the HD-vectors representing the entities. A bit-wise thresholded sum of n vectors results in 0 when n/2 or more arguments are 0, and 1 otherwise. Furthermore terms "thresholded sum" and "majority sum" are used interchangeably and denoted as [A+B +C]. The relevant properties of the majority sum are: • the result is a random vector, i.e. the number of '1' components is approximately equal to the number of '0' components; • the result is similar to all vectors included in the sum; • the number of vectors involved into MAJORITY sum must be odd; • the more HD-vectors that are involved in a majority operation, the closer the Hamming distance between the resultant vector and any HD-vector component is to 0.5; and • if several copies of any vector are included into a majority sum, the resultant vector is closer to the dominating vector than to other components.
The algebra on VSA includes other operations e.g., binding, permutation [4]. Since we do not use them in this article, we omit their definitions and properties.

IV. HOLOGRAPHIC GRAPH NEURON
This section presents one of the main contributions of this paper: the adoption of the VSA data representation for implementation of the HGN approach.

A. Motivation for Holographic Graph Neuron and outline of the solution
An important issue in hierarchical models is the overhead of resource requirements, specifically with regards to the number of processing elements required. We propose a holographic approach, which enables a flat GN array to operate with higher level of accuracy and comparable recall time than that of HGN, without the need for a complex topology and additional nodes. The high-level logic of the proposed solution is illustrated in Figure 2. Essentially, we replace the need for maintaining topological relationships by compacting parts of the pattern observed by GNs into one holographic representation.

B. Encoding
In the case of HoloGN, all its elements (i.e. symbols of individual neurons) are indexed uniquely and the index of a particular element is derived as a function of the GN's ID. Let IV j be an initialization high-dimensional vector for GN j. The initialized vectors for different GNs are chosen to be mutually orthogonal. Then the HD-index of element i in GN j is computed as E HD (j,i) = Sh(IV j , i), where Sh() is a cyclic shift operation resulting in the generation of a vector orthogonal to IV j HD-vector [17], [18].

C. Construction of VSA-representation of activated GNs
Let n be the number of individual Graph Neurons. When a GN array is exposed to a pattern, the activated elements communicate their HD-represented indices to all other GNs; the holographic representation of the activated elements is then: where E HD j is the HD-index of the activated element in textGN j , and the addition operation is the bundling operation, or thresholded sum as described in III-B. As discussed previously, in the resultant HD-vector the Hamming distances between each component and the composition vector is strictly less than 0.5. We will utilize this property later when constructing data structures for recall of patterns in HoloGN.

D. Data structures for storing and retrieving holographic representations in HGN elements
HoloGN will store the holographic representation of the entire pattern (3) observed across all GNs. A possible architecture is for all memorized patterns to be collected and stored centrally at a processing node.
Depending on the particular application of HoloGN, the memorized patterns could be stored either in an unsorted list or in bundles. The first mode of storing HoloGN patterns corresponds to the case where the structure of the observed patterns is unknown; the latter mode is used in the case of a supervised learning. The next section describes different HoloGN usage and recall strategies.

V. HOLOGN RECALL STRATEGIES
This section introduces and evaluates the performance of two major recall modes: the one-shot case and the case of supervised learning.
A. Time-efficient ξ-accurate recall in an unsorted HoloGN storage The common steps in both recall modes is the procedure for the time-efficient search over an (unsorted) list of HoloGN records. Recall that all manipulations with VSAencoded entities are done using simple bit-wise arithmetic operations as well, and calculations to obtain the Hamming distance between entities. However, for this article we assume no particular optimized implementation of VSA's bit-wise operations; this is because such operations are tailored to the architectures of specific microprocessors, which operate with words of substantially lower dimensionalities (typically 32 or 64 bits). Therefore, adopting these methods for implementing the bit-wise operations on words of thousands of bits would be cumbersome. Instead, an easily analyzable computational model is adopted in this article, which could also be adapted to an implementation on specialized computing architectures.
In what follows, each HoloGN pattern h i is modelled as a row vector of d elements. The list of stored HoloGN patterns is therefore modelled as an l ×d matrix, where l is the number of the learned (stored) HoloGN patterns. Denote this matrix as H. The task of recalling a pattern with a target accuracy of ξ (ξ < 0.5) is formulated as finding the rows h i in H with Hamming distance to the query pattern h q less than or equal to ξ.
The conventional way of computing Hamming distance between vectors would be to perform the following sequence of computations for each row in H: 1) perform an elementwise XOR with the vector query; 2) sum up all elements in the intermediate result; and then 3) divide the result by the dimensionality of the vectors. The performance of the two implementations of this method using Matlab's repmat and bxf un functions are shown by the top two curves in Figure 3. The curves demonstrate linear but rapid increase in the recall time with the increase in the number of the stored patterns. The lowest curve in the figure shows the performance of matrixvector multiplication of the same size, which is chosen as the reference case. The results were obtained on a Intel Core i7-3520M 2.9 GHz machine with Windows 7 operating system using one processor.
1) Binary spatter codes as complex numbers: In order to improve the efficiency of calculating Hamming distances over a vast number of HoloGN patterns, we propose representing HoloGN patterns using complex numbers, where a binary 0 would be represented by √ −1 (i.e. the complex number) and a binary 1 would remain 1. The intuition behind this transformation is simple: multiplication of bits in the same position should produce three outcomes: −1 = j ×j, 1 = 1×1 and j = 1 × j. That is, the multiplication of two similar bits would produce a real number and the multiplication of two different bits produces a imaginary number. In this way the sum of the imaginary parts over all positions in the resulting vector, divided by dimensionality d, will correspond to the Hamming distance between the two vectors. Thus, the suggested method allows us to implement the calculation of Hamming distance through the standard method of matrixvector multiplication, as illustrated below: The performance of the proposed method is illustrated by the red dashed curve in Figure 3. It demonstrates that the calculation of the Hamming distance is only two times slower than the usual matrix-vector multiplication. Specifically, to compute Hamming distance from the target vector to each of 20 000 stored patterns takes approximately 200 ms on the test machine. Further optimization of the matrix multiplication and execution on parallel architectures hint on realistic bounds on the recall time over extremely large numbers of patterns.

B. Case-study 1: Best-match probing under one-shot learning
The first usage of HoloGN is probing the existence of the target query pattern amongst the previously memorized patterns. The perfect match in this case would be indicated by a Hamming distance of zero. The deviation from zero, therefore, reflects the degree of proximity of the query to one or several stored HoloGN patterns. In the following example the accuracy of the HoloGN recall was compared to the performance of the original HGN approach. For the sake of fair comparison the 7 by 5 pixels letters of the Roman alphabet (as in [2]) were used. In the memorizing phase a set of noise-free images of letters illustrated in Figure 4 was presented to both architectures. In the recall phase images of the same letters distorted with different levels of random distortions (between 1 bit corresponding to a distortion of 2.9% of the pattern's size and 5 bits equivalent to 14.3% distortion) were presented to the architectures for the recall. An example of a noisy input is presented in Figure 5. In the case of HoloGN the pattern with the lowest Hamming distance to the presented distorted pattern was returned as the output. Figure 6 presents the results of the accuracy comparison between the recall results for the HoloGN approach and the reference HGN architecture. To obtain the results 1000 distorted images of each letter for every level of distortion were presented for recall. The charts show the percentage of the correct recall output. The analysis shows that the performance of the HoloGN-based associative memory at least matches that of the original approach. In most of cases HoloGN appears to be more accurate when recalling patterns with high level of distortion.

C. Case-study 2: HoloGN recall under the supervised learning
The analysis presented above is a very positive result for the proposed bio-inspired associative memory based pattern processing architecture, since the accuracy of the original HGN approach was demonstrated to be as accurate as artificial neural networks with back-propagation [2]. While establishing formal relationships to the framework of artificial neural networksis outside the scope of this work, this section presents the results of the pattern recognition accuracy of the HoloGN architecture under the supervised learning. In this case the HoloGN is presented with a series of randomly distorted patterns for each letter with different level of distortion (between 3% and 43%) as exemplified in Figure 5. In the experiments up to 500 patterns for each letter and every level of distortion were presented for memorizing. For the particular level of distortion i all e presented patterns of the particular letter L i were bundled to a single HoloGN representation as Thus by the end of the learning phase the HoloGN list will contain 26 high dimensional bundles, each jointly representing all (presented) distorted variants of the particular letter. In the recall phase for each distortion level HoloGN was presented with 500 new distorted patterns of each letter. The accuracy of the recall was measured as the percentage of the correctly recognized letters averaged over the alphabet. Figure 7 illustrates the obtained results: 90% accurate recall was observed when learning symbols distorted up to 15%. While the accuracy predictably decreases rapidly with the increase of distortion in the presented patterns, a reasonable 80% recall accuracy was observed for learning sets with 7 bits distortion (20%). Figure 8 illustrates the convergence of the HoloGN recall accuracy with the number of presented noisy samples for the case of 14.3% distortion (5 bits). This is a definitely positive result for the presented architecture, which illustrates the suitability of the HoloGN in applications requiring supervised learning.

ANALYSIS
There is a class of pattern recognition applications which requires an understanding of the details of the recall results. For example, when a recall returns several possible patterns of given recall accuracy, the task would be to understand the overlapping elements. This section considers two aspects of this task: a robust decoding of elementary components out of a distributed VSA representation; and a quantitative metric of the similarity via direct comparison of distributed representations, without the need for decoding those representations.
The VSA approach of representing data structures by definition makes decoding of the individual components a tedious task, requiring a brute force test on the inclusion of all possible high-dimensional codewords for each GN. The majority sum, which is used for creating HoloGN representations of the observed patterns imposes a limit on the number high-dimensional codewords operands, above which a robust decoding of the individual operands is impossible.

A. Preliminaries
Denote the density of a randomly generated HD-vector (i.e. the number of ones in a HD-vector) as k. The probability of picking a random vector of length d with density k, where the probability of 1's appearance, defined as p, is described by (2). The mean density of a random vector is equal to d·p. Note that in reality the density of randomly generated HD-vectors will obviously deviate from the mean value. However, according to (1) density k is approaches the mean value with the increase of dimensionality d. In other, words the probability of generating HD-vector with k >> d · p or k << d · p decreases with the increase of dimensionality d. Define thr as the threshold probability of generating a vector with a certain deviation of density being negligibly small. Let k − and k + characterize the lower and the upper bounds of the interval of possible densities. This is illustrated in Figure 9. The bounds for a given d, p and thr are calculated using (2). The bounds are calculated according to (6) and (7). The value of threshold is chosen to be small (10 −6 ).

B. Capacity of HoloGN representations
Suppose there exists an item memory [4] containing HDvectors representing atomic concepts 3 . Recall that when several HD-vectors are bundled by the majority sum the noise of flipped bits increases with the number of components. For a given dimensionality d there is a limit on the number of bundled HD-vectors beyond which the resulting HD-vector becomes orthogonal to every component vector; hence the A capacity of the resulting vector is defined as the maximal number of mutually orthogonal HD-vectors which can be robustly decoded from their majority sum composition by probing the item memory.
In order to characterize the capacity of the composition vector for a given dimensionality, one needs to characterize the level of noise p n introduced by the bundling operation. This is calculated as in [19] by (8), where n is a number of atomic vectors in the resulting majority sum vector.
Consider an arbitrary HD-vector A to be decoded from a majority sum composition. Let N be a vector of noise imposed by the majority sum operation. Since the components 3 In the case of HoloGN an atomic concept is the code for the particular HoloGN element. are mutually orthogonal, the density of ones in the noise vector is also described the binomial distribution Pr(k, d, p n ). As each new vector is added the level of noise increases, hence the mean of the noise vector density will approach 0.5 as illustrated in Figure 11. Due to the properties of high dimensional space, vector A will be undecodable when the upper bound k + (d, p n , thr) of the density of noise vector N approaches the lower bound k − (d, 0.5, thr). That is, the noisy version of A becomes orthogonal to its clean version. This logic is illustrated in Figure 10, where the resulting majority sum vector is orthogonal to all components. Thus the capacity of the distributed representation with dimensionality d is computed by: Capacity(d, thr) = = max n (k + (d, p n (n), thr) ≤ k − (d, 0.5, thr)) (9) Fig. 11. Capacity of the component vector versus the dimensionality. The calculations use a the threshold of thr = 10 −6 Figure 11 presents the capacity of HD-vector of different dimensionalities calculated for threshold probability thr = 10 −6 . Specifically for d = 10000 bits the capacity of the robustly decodable VSA is 89 vectors.

C. Calculation of the number of common component vectors in two resulting vectors
Given the rather conservative limits on the number of robustly decodable elements in a distributed representation it is important that the proposed HoloGN architecture can estimate the similarity between different patterns without decoding them. This subsection provides a method for the quantitatively measuring the number of overlapping elements as a function of their relative Hamming distance. Denote m and n as lengths of two patterns, m <= n, and denote c as the number of common elements in these patterns. Let M be a c × d matrix of common elements where each row contains a random HDvector of dimension d encoding element c i . Denote an arbitrary column of matrix M as C. Since rows in M are independent, the density of ones in each column also follows the binomial distribution with p = 0.5 and length c. Denote the number of ones in column C as ||C|| 1 .
In order to calculate the Hamming distance between the distributed representations of two patterns with known m, n and c, consider all possible cases when bits in the same position are different. The Hamming distance between two patterns can be estimated with equation (10): ·p 0 (n, c, ||C|| 1 )+p 0 (m, c, ||C|| 1 ) · p 1 (n, c, ||C|| 1 )); (10) where p i (j, c, ||C|| 1 ) stands for the probability of having i (0 or 1), when the representation consists of j = m or j = n atomic vectors and c of these vectors are overlapped.
Due to the symmetry in the calculation of probabilities, p i (j, c, ||C|| 1 ) is presented only for the case of p 1 (m, c, ||C|| 1 ). There are three possible cases for calculation of p 1 (m, c, ||C|| 1 ): • if ||C|| 1 is more than m/2, then the result of the majority sum is '1', i.e. p 1 is 1; • if number of possible '1's is less than m/2, then probability of p 1 is 0; • otherwise the probability should take into account all possible combinations, and their probabilities. Equation (11) specifies probability for p 1 (m, c, ||C|| 1 ), and equation (12) does the same for p 0 (m, c, ||C|| 1 ).
p 0 (m, c, Figure 12 shows the Hamming distances between two resulting vectors for different numbers of overlapping vectors. The results show that the larger the number of common elements, the smaller the Hamming distance between resulting vectors.
This method opens a way towards constructing and analyzing patterns far beyond VSA's robustly decodable capacity. The problem with practical application of this method, however, comes with the rapid convergence of the Hamming distance indicator to 0.5, making the difference between analyzable patterns indistinguishable as illustrated in Figure 12. For example, for HoloGN representations of patterns with 15 elements, sub-patterns of 3 overlapped elements are robustly detected, while patterns with fewer overlapped elements are indistinguishable. Thus, the minimal number of overlapped elements in two patterns which can be robustly detected using the Hamming distance indicator is called bundle's sensitivity.
The analysis of the sensitivity is similar to the analysis of the capacity of VSA representation in section VI-B. For two patterns of length m and n elements, and c overlapped components, the sensitivity is calculated by (13): Sensitivity(d, thr, m, n) = min c (k + (d, p(c, n, m), thr) <= k − (d, 0.5, thr)) (13) Figure 13 demonstrates the development of the sensitivity threshold with the number of elements in the compared patterns. The results show that the number of components for robust detection grows linearly with the size of pattern. Patterns with more than 500 elements should contain at least 14 % overlapped elements to be robustly detected by the proposed method.

VII. CONCLUSION
This article presented Holographic Graph Neuron -a novel approach for memorizing patterns of generic sensor stimuli. HoloGN is built upon the previous Graph Neuron algorithm and adopts a Vector Symbolic representation for encoding of the Graph Neuron's states. The adoption of the Vector Symbolic representation ensures a one-layered design for the approach, which implies the computational simplicity of the operations. The presented approach possesses the number of uniques properties. Prior to all, it enables a linear (with respect to the number of stored entries) time search for an arbitrary sub-pattern. While maintaining the previously reported properties of Hierarchical Graph Neuron, HoloGN is also improving the noise resistance of the architecture by this substantially improving the accuracy of pattern recall.