{'fraction_non_alphanumeric': 0.0429375054935396, 'fraction_numerical': 0.015645600773490375, 'mean_word_length': 4.203521609878802, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bi-stability of mixed states in neural network storing hierarchical patterns

We discuss the properties of equilibrium states in an autoassociative memory model storing hierarchically correlated patterns (hereafter, hierarchical patterns). We will show that symmetric mixed states (hereafter, mixed states) are bi-stable on the associative memory model storing the hierarchical patterns in a region of the ferromagnetic phase. This means that the first-order transition occurs in this ferromagnetic phase. We treat these contents with a statistical mechanical method (SCSNA) and by computer simulation. Finally, we discuss a physiological implication of this model. Sugase et al. analyzed the time-course of the information carried by the firing of face-responsive neurons in the inferior temporal cortex. We also discuss the relation between the theoretical results and the physiological experiments of Sugase et al.


I. INTRODUCTION
There are two kinds of hypotheses regarding internal representations of memory items in the brain. First, there is the "distributed representation" hypothesis, which assumes that our memory items are encoded by neuron activity patterns. But another hypothesis (e.g., the "grand mother cell" hypothesis) has been proposed, where the memory items are represented by the excitation of corresponding neurons: that is, the "local representation" hypothesis. Let us consider a neural network model consisting of N neurons. The number of memory items in the local representation is O(N). One might think that the number of memory items in the distributed representation is O(2 N ) in the case of storing binary patterns. However, this consideration is meaningless. If we consider error correcting abilities, this becomes O(N) on the basis of the statistical mechanical theories (for example [2]). One of the remarkable advantages of the distributed representation is that the relationship between the memory items is naturally implemented by the distance of memory patterns. However, many studies on associative memory models have been confined to those with uniformly distributed patterns. Thus, we should discuss a model that stores some structural patterns. We will extend the conventional SCSNA [3] to a generalized SCSNA in order to treat a model with any structural pattern and a general class of output functions of neuron.
When an associative memory model is made to store memory patterns as a result of correlation learning, a pattern, which has an uniform overlap with some of the stored memory patterns, automatically becomes the equilibrium state of the model. This is called the mixed state. It is not appropriate to think that this mixed state is a side-effect and/or that it is unnecessary for information processing. Amari has discussed a "concept formation" using the stability of mixed state [4]. The correlated attractor [5,6] accounting for the physiological experiments by Miyashita [7] can be interpreted as a mixed state in a broad sense. Recently, Parga and Rolls [8] used the mixed state in their research on the mechanism of invariant recognition with a coordinate transformation in the visual system.
The aim of this research is to study the properties of mixed states in the autoassociative memory model storing hierarchical patterns by utilizing the generalized SCSNA and computer simulation. First, we discuss a model that stores patterns in which a two-stage hierarchy exists. With the generalized SCSNA, we derive the order parameter equations for the equilibrium state in this model. By solving the obtained equations, we show that two kinds of mixed states coexist in a particular region of the retrieval phase (ferromagnetic phase). It is a characteristic of the two kinds of mixed states that they have different values of cross-talk noise variances, that is, they are influenced by the uncondensed patterns in different ways. This kind of bi-stability in the retrieval phase has not been previously reported. We will show that the bi-stability of mixed states does not depend on the number of patterns in the same cluster. Generally speaking, it is not so easy to confirm the multi-stable states by computer simulation. However, we succeed in confirming the bi-stability of the mixed states by computer simulation after considering the qualitative properties of the retrieval dynamics in this model. Next, we treat a model storing a set of hierarchical patterns and uniformly distributed patterns in order to investigate universality of the bi-stability. From these theoretical results, we found that a bi-stability of mixed states also exists in this model. Contrary to these models, such bi-stability of mixed states does not exist in a model where the contribution of uncondensed patterns to the synaptic couplings is replaced by the spin glass type interaction. We will discuss the reason for this by using the SCSNA.
Recently, Sugase et al. have reported interesting phenomena concerning the temporal dynamics of face-responsive neurons in the inferior temporal (IT) cortex [1]. In the discussion, we will examine the relation between the obtained results and the physiological experiments by Sugase et al.

II. MODEL
Let us consider a recurrent neural network consisting of N neurons with an output function F (·). We employ the synchronous dynamics, where x t i represents a state of the i-th neuron at discrete time t, and discuss the case of the thermodynamics limit (N → ∞). J ij in the above equation denotes a synaptic coupling from the j-th neuron to the i-th neuron. In this work, we discuss the equilibrium state of eq. (1). For simplicity, we treat a two-stage hierarchy, which is one of the simplest cases. This can be easily extended to more complex hierarchies. One can use many procedures to make the set of ultrametric patterns, but we employ the following method. Each component ξ µ,ν i of pattern ξ µ,ν is a random variable drawn from the following probability distributions, with 0 ≤ b ≤ 1. The distance between patterns ξ µ,ν is expressed by where E[·] stands for an average with respect to the probabilities distributed in eqs. (2) and (3), and δ µµ ′ is the Kronecker's δ defined as, According to the definition of the matrix B in eq. (5), B is s × s matrix. As shown in eq. (4), the memory patterns ξ µ,ν have a two-stage ultrametric structure. (B) νν ′ in eq. (4) stands for the element in the ν-th row, the ν ′ -th column of matrix B. ξ µ,ν are ps uniformly generated patterns when b = 0, while ξ µ,ν in the µ-th cluster are the same when b = 1.
We employ the simple Hebbian rule as the learning rule, and the synaptic coupling J ij is set to where α = p/N. Since the number of clusters is αN, we call α the loading rate. ξ µ is not explicitly used for the learning rule. We discuss two kinds of models. One is the previously defined model with the couplings J ij given by eq. (7), which we call Model 1. The other model, Model 2, is shown here with the following couplings J ij , We will examine the stability of the memory pattern and a mixed state which has a uniform overlap with an odd number of memory patterns in the same cluster. There exist mixed states which have finite overlaps with memory patterns in different clusters. However, we do not discuss this topic in the present paper.

III. A GENERALIZED SCSNA
There are two kinds of statistical mechanical theories treating the equilibrium properties of associative memory models. One is based on the SCSNA proposed by Shiino and Fukai [3]. The other one is the replica theory, which has been used to analyze the equilibrium states in Model 1 [9]. However, the replica theory cannot be applied to a system where the free energy cannot be defined (e.g., a model with a nonmonotonic output function [10] or an oscillator associative memory model [11]), while the SCSNA can treat these systems without the energy function. Since previous studies with the SCSNA have mainly focused on systems with uniformly distributed patterns, the SCSNA cannot be directly applied to Model 1.
We will extend the previously proposed SCSNA to a generalized SCSNA in this section in order to treat a model storing a set of structural patterns σ ρ (ρ = 1, 2, ...,αN) randomly generated by an independent identical probability distribution with respect to i. The correlation matrix (C) ρρ ′ between the two patterns σ ρ and σ ρ ′ is given by, We consider a recurrent network consisting of N neurons with the output function F (·) and synaptic couplings J ij given by Since the number of memory patterns isαN,α is defined as the loading rate in this model. ξ µ,ν or B in eq. (4) is an example of σ ρ or C in eq. (10), respectively, We consider the case where the state of neurons at discrete time step t, x t , is synchronously updated, and discuss the equilibrium state x with the limit t → ∞. We introduce a set of rotated memory patterns,σ = {σ i 1 ,σ i 2 , ...,σ iα N }, as w ρ (ρ = 1, 2, ...,αN) and κ ρ (ρ = 1, 2, ...,αN) in the above equations represent the ρ-thαN dimensional normalized eigenvector of matrix C and the ρ-th eigenvalue of matrix C for w ρ , respectively. Each componentσ ρ i is statistically dependent (independent of) on µ (i), respectively, and satisfies the following conditions, Using the rotated patterns, we can rewrite J ij in eq. (11) as The overlapsm ρ between the equilibrium state x andσ ρ are defined by the following equation,m If one assumes that the equilibrium state x has nonzero overlaps withs rotated patterns σ ρ (1 ≤ ρ ≤s), we can derive the SCSNA order parameter equations [12] (see Appendix A), where < · · · >σ stands for an average over the condensed patternsσ = (σ 1 ,σ 2 , ...,σs). We can express the sum of κ ρ in terms of an integration along continuous eigenvalue κ( ρ αN ) ≡ κ ρ for p, N → ∞. We remarked that the analytical expressions of r and Γ given by eqs. (27) and (28) depend only on the matrix C and do not explicitly depend on the condensed patternsσ ρ (ρ = 1, 2, ...,s). This fact leads to the following order parameter equations for the equilibrium state having nonzero overlaps Note that the off-diagonal terms of the matrix C between the condensed and uncondensed pattern spaces can be neglected ifs is taken to be sufficiently large but O(1).

IV. RESULTS
We discuss the case where F (·) = sgn(·) defined by First, we will show the results of the analysis for the equilibrium properties in Model 1. Now, let us apply the generalized SCSNA proposed in section III to Model 1. We rewrited eq. (13) by The matrix B has the eigenvalues . By considering the relations given by eqs. (12), (14) and (37), we obtained the following SCSNA order parameter equations for this model: where < · · · > stands for an average over the condensed patterns ξ 1 = (ξ 1,1 , ξ 1,2 , · · · , ξ 1,s ).
In the above equations, we omitted the Γ terms corresponding to eq. (35) by applying the Maxwell rule. The results coincided with those of the theoretical analysis using the replica theory [9]. For simplicity, we will show the case where s = 3, 5. Fig. 1 shows a phase diagram of numerical solutions for eqs. (38) -(41). The critical loading rate of a memory pattern and that of a mixed state are plotted against b. The triangular area shown in the figure represents the region where two kinds of mixed states coexist. We called this region the "bistable region". In this paper, a mixed pattern which is similar to sgn( s ν=1 ξ µ,ν ), is defined as η µ , and the other one, which is reported here for the first time, is defined asη µ . As shown in [3], the SCSNA assumes the stability of equilibrium state. There is no free energy in the SCSNA formalism. However, we discuss the present model using the equilibrium statistical mechanics [9] since it has the energy function. From the statistical-mechanical viewpoint, the first-order transition occurs at the dashed curve in Fig. 1. In the region below the dashed curve, the free energy of η µ is less than that ofη µ , while the free energy of η µ is larger than that ofη µ in the region above the dashed curve.
We will examine two typical examples in the bi-stable region. We consider the case where the retrieval pattern is ξ 1,1 . α η c (αη c ) is defined as the critical loading rate of η 1 (η 1 ). First, we discuss the region in which α η c < αη c as follows. Fig. 2(a) shows how the overlaps between the equilibrium state and the retrieval pattern ξ 1,1 depend on α. Two kinds of mixed states coexist when 0.01500 < α < 0.01765, as shown in Fig. 2(a). From the numerical analysis, the value of the cross-talk noise variance expressed by r in eq. (40) becomes larger in the order with α, as shown in Fig. 2(a). Thus, we found thatη 1 is more influenced by the uncondensed patterns than η 1 is. We will transform the the order parameter equations given by eqs. (38) -(41) into an equation in one variable in order to qualitatively understand the nature of this bi-stability. Considering that m 1,ν is the same value for any ν in the mixed states, we can transform the order parameter equations representing the mixed states into the equation in one variable y by replacing m 1,ν √ 2αr with y ν (see Appendix B). Fig. 2(b) shows the graphical solutions of the transformed equation around the bi-stable region. Each intersection of the y-axis and function Φ(y, α, b = 0.61, s = 3) represents the corresponding solution. Fig. 2(b) shows that two kinds of mixed states, that is, η 1 andη 1 , coexist at α = 0.016. On the other hand, there is no such bi-stability in the retrieval region except for the triangular region. Fig. 1 shows that bi-stability also exists for s = 5. By analyzing the graphical solutions of the transformed equation for various values of s, we deduced that the bi-stability of the mixed states exists for any value of s in Model 1.
We performed computer simulation to confirm the bi-stability of mixed states, and obtained equilibrium states, which are expressed by the dotted curve (A) in Fig. 2(a), as follows. We first got an equilibrium state at α = 0.0003 < α η c with the initial state set to sgn( s ν=1 ξ 1,ν ). After that, equilibrium states for various α were obtained by gradually increasing the value of α from α = 0.0003. Since the simulation results mostly agreed with the theoretical results, we could not distinguish between them as shown in this figure. The dotted curve (B) in Fig. 2(a) was obtained as follows. An equilibrium state at α = 0.019 (α η c = 0.01765 < α < αη c = 0.01982 ) was obtained by gradually increasing the value of α from α = 0.0003 in computer simulation. Using this equilibrium state, we got each equilibrium state at the corresponding α by gradually decreasing the value of α from α = 0.019. We carried out computer simulation with N = 10000; a typical example is shown in Fig. 2 (a). Fig. 3(a) shows a typical example (b = 0.55, s = 3) of the case where αη c < α η c . Two kinds of mixed states coexisted when 0.01164 < α < 0.01389. The axes in this figure are the same axes in Fig. 2(a). Fig. 3(b) shows the graphical solutions of the transformed equation in one variable around the bi-stable region when s = 3, b = 0.55. This figure shows that two kinds of mixed states, that is,η 1 and η 1 , are stable at α = 0.0125. The dotted curve (C) in Fig. 3(a) was obtained as follows. We first got an equilibrium state at α = 0.0003 with the initial state set to sgn( s ν=1 ξ 1,ν ). After that, equilibrium states for various α were obtained by gradually increasing the value of α from α = 0.0003. Since η 1 is always stable when αη c < α < α η c , an equilibrium state corresponding toη 1 could not be obtained by computer simulation with the initial state set to sgn( s ν=1 ξ 1,ν ). By considering the qualitative properties of the retrieval dynamics, we carried out computer simulation in the following manner. Fig. 4 shows the trajectories of temporal evolutions of overlaps of the state x t with ξ 1,1 or ξ 1,2 respectively, The parameters were set to s holds over the retrieval process in this case. crosses that are shown in Fig. 4 represent the theoretically obtained equilibrium states corresponding toη 1 , η 1 and ξ 1,1 . Two kinds of mixed states (η 1 ,η 1 ) and the retrieval pattern (ξ 1,1 ) were stable at this parameter set ( Fig.  4). The dynamics converged toη 1 , η 1 , ξ 1,1 in order as m 1,1 0 increased, while the cross-talk noise variance for the equilibrium became larger in the order, ξ 1,1 , η 1 ,η 1 . This phenomenon occurred because the cross-talk noise variance of the initial stage of the retrieval process increased as m 1,1 0 decreased. The results showed that equilibrium states corresponding toη 1 can be confirmed by computer simulation when the initial state has an appropriate overlap with the retrieval pattern. The dotted curve (D) in Fig. 3(a) was obtained as follows. We got an equilibrium state at α = 0.0135 where the initial state had the appropriate overlap with the retrieval pattern (ξ 1,1 ). We obtained equilibrium states for various α by increasing (decreasing) the value of α from α = 0.0135. We carried out computer simulation with N = 10000; Fig. 3(a) shows a typical example. Although the result from this simulation did not always quantitatively agree with the theory owing to spurious states, we can state that the bi-stability of two kinds of mixed states exists in the particular region of the retrieval phase from the hysteresis shown in Figs. 2(a) and 3(a). Let us present the equilibrium properties in Model 2. Since the uncondensed patterns ξ µ (µ = 2, 3, ..., αN) of Model 2 satisfy the orthogonal condition E[ξ µ i ξ µ ′ i ] = δ µµ ′ , the order parameter equations for this model were derived by replacing eq. (40) with eq. (45): We transformed the order parameter equations concerning the mixed states into an equation in one variable y ν = m 1,ν √ 2αr (see Appendix B). As in Model 1, we deduced that there is bistability of mixed states for an arbitrary s. These results indicated that this bi-stability for mixed states is not dependent on details of the structure in stored patterns.

V. DISCUSSION
We have shown that bi-stability of mixed states exists in two typical models, Model 1 and Model 2. The reason for this bi-stability, which is based on the SCSNA, can be explained as follows. We introduce a model where the contribution of uncondensed patterns to the coupling J ij is replaced by the spin glass type interaction, and we call this model Model 3. The synaptic coupling of Model 3 is defined as follows, and the symmetric noise δ ij is independently drawn from the following rule, where δ is constant. We obtained the following order parameter equations for this model with the SCSNA [13], As you can notice, this model corresponds to the Sherrington-Kirkpatrick (SK) model. In contrast to Model 1 or Model 2, a bi-stability of mixed states does not exist in Model 3. Note here that the analytical expression of the cross-talk noise variance expressed by eq. (40) or eq. (45) is explicitly expressed as a function of susceptibility U, which is a kind of a cross-talk noise-enhancement factor caused by the full feedback nature of the model. However, the analytical expression of the cross-talk noise variance for Model 3, σ in eq. (56), is not dependent on the susceptibility U. The above results show that the bi-stability is due to not only the hierarchy of stored patterns but also analytical expression of the cross-talk noise variance, which is a function of susceptibility for the model. Naturally, the bi-stability may exist in more complex hierarchies because the corresponding cross-talk noise variance is explicitly expressed by a function of susceptibility as shown in eq. (34). From these results, we also found that the bi-stability of mixed states does not exist in a strongly diluted system [14]. Finally, we discuss the relation between findings concerning the retrieval process in Fig.  4 and the physiological experiments of face-responsive neurons in the IT cortex by Sugase et al [1]. Recently, Sugase et al. have analyzed the time-course of information carried by the firing of face-responsive neurons in the IT cortex, while performing a fixation task of monkey and human faces with various expressions, and simple geometrical shapes. They found that the initial transient firing correlated well with a rough categorization (e.g., face vs. non-face stimuli). Their results suggest that the neuron firing pattern is initially a superposition of patterns representing different faces or expressions, but it then converges to a single pattern representing a specific face or expression. We found that the retrieval dynamics of Model 1, shown in Fig. 4, can qualitatively replicate the temporal dynamics of face-responsive neurons as follows [15]. Initially, the network state approaches a mixed state (η 1 or η 1 ) that is a superposition of patterns representing different persons or expressions. After that it diverges from the mixed state, and finally converges to a single memory pattern (ξ 1,1 ) representing a specific person or expression as shown in Fig. 4. From the above results, we expect that the present system may mimic the temporal dynamics of the face-responsive neurons [15]. Details will be discussed elsewhere.

ACKNOWLEDGMENTS
This work was supported in part by Grants-in-Aids #09308010 and #11145229 for Scientific Research from the Ministry of Education, Science, Sports and Culture of Japan. We are indebted to Tomoki Fukai for useful discussion.

APPENDIX A:
The internal potential of the i-the neuron h i in the equilibrium state of eq. (15) is The output x i can be formally expressed as where the functionF (·) will be determined later as shown in eq. (26). Here, we assume that the equilibrium state x has nonzero overlaps withs rotated patternsσ ρ (ρ = 1, 2, ...,s). The residual overlapm ρ ∼ O(1/ √ N ), (s + 1 ≤ ρ ≤αN) can be derived by using Taylor expansionm Substituting eq. (A7) into eq. (A2), we obtain where Γ is defined as andz i is the effective noise so that, Note thatz i is a summation of uncondensed patterns with <z >= 0 and <z 2 >=αr, Let us express the sum of κ ρ in terms of an integration along continuous eigenvalue κ( ρ αN ) ≡ κ ρ for p, N → ∞, We can obtain eqs. (22) -(28) by replacing x i with Y .

APPENDIX B:
Let us transform the order parameter equations for the mixed state in Model 1 into the equation in one variable. We introduce y ν in the following equation, Since y ν = y (ν = 1, 2, 3, ..., s) holds in the mixed state, we can express the order parameter equations for the mixed state by the following equation in y, where λ ν is the ν-th eigenvalue of the matrix B. The order parameter equations for the mixed states in Model 2 can be also transformed into the equation in one variable as follows,