# Mutual information and self-control of a fully-connected low-activity neural network

## I Introduction

It is well-known by now that low-activity neural network models have a larger storage capacity than the corresponding models with a mean 50% activity (see, e.g., ). However, this improvement is not always apparent in the basins of attraction. Furthermore, for low activities the information content in a single pattern is reduced. For these reasons it is argued that a neural activity control system is needed in the dynamics of the network in order to keep its activity the same as the one for the memorized patterns during the whole retrieval process . Recently, new suggestions have been put forward for the choice of threshold functions in network models in order to get an enhanced retrieval quality– overlap, basin of attraction, critical capacity, information content (see  and references therein). Diluted models , layered models  and models for sequential patterns  have been considered. In all cases it has been found that appropriate thresholds lead to considerable improvements of the retrieval quality.

The models mentioned above have a common property. For the diluted and layered models there is no feedback in the dynamics. For the model with sequential patterns no feedback correlations are taken into account. The absence of feedback considerably simplifies the dynamics. Hence, it is interesting to look at a model with feedback correlations and to see whether the introduction of a threshold in the sense described above still enhances the retrieval properties in this much more complex situation.

With these ideas in mind we consider in the sequel low activity (or in other words sparsely coded) fully connected neural networks. In particular, we study the application of a self-control mechanism proposed recently for a diluted network of binary patterns . Self-control has been introduced in order to avoid imposing some external constraints on the network with the purpose of improving its retrieval properties. Such external constraints destroy the autonomous functioning of the network.

The model we look at is a fully-connected attractor neural network with neurons and patterns taking the values $`\{1,0,+1\}`$ and pattern activity $`a`$. A low-activity neural network corresponds then to the case where the pattern distribution is far from uniform, i.e., $`a<2/3`$. This network has the advantage that it can be generated keeping a symmetric distribution of the states since both the $`\pm 1`$ states are considered the active ones, while the $`0`$ state is the inactive one.

The rest of this paper is organised as follows. The three-state network model and its order parameters are described in section II. In order to study the retrieval quality of the model, especially in the limit of low activity the mutual information content is analysed in Section III. Section IV discusses the dynamics of this network in the presence of the self-control mechanism realised through the introduction of a time-dependent threshold. Evolution equations for the order parameters are written down. Using these equations the influence of self-control on the retrieval quality of the network – information content, critical capacity, basins of attraction – is studied in section V. Furthermore, these theoretical findings are compared with results from numerical simulations of a fully connected network of $`10^4`$ neurons. Finally, section VI presents some concluding remarks.

## II The model

Consider a neural network model of $`N`$ three-state neurons. At a discrete time step $`t`$ the neurons $`\sigma _i\{0,\pm 1\},i=1,\mathrm{},N`$ are updated according to the parallel deterministic dynamics

$$\sigma _{i,t+1}=F_{\theta _t}(h_{i,t}),h_{i,t}=\underset{i(j)}{\overset{N}{}}J_{ij}\sigma _{j,t}$$
(1)   
where $`h_{i,t}`$ is the local field of neuron $`i`$ at time $`t`$ and $`\theta _t`$ a time-dependent threshold parameter. As usual, the transfer function $`F_{\theta _t}`$ is given by

$$F_{\theta _t}(x)sgn(x)\mathrm{\Theta }(|x|\theta _t)$$
(2)   
with $`\mathrm{\Theta }`$ the standard Heaviside function.

The couplings $`J_{ij}`$ are determined as a function of the memorized patterns $`\xi _i^\mu `$ by the Hebbian learning algorithm

$$J_{ij}=\frac{1}{Na}\underset{\mu =1}{\overset{p=\alpha N}{}}\xi _i^\mu \xi _j^\mu $$
(3)   
with $`\alpha `$ the loading capacity. The patterns are taken to be independent identically distributed random variables (IIDRV) $`\xi _i^\mu \{0,\pm 1\},i=1,\mathrm{},N,\mu =1,\mathrm{},p`$, chosen according to the probability distribution

$$p(\xi _i^\mu )=a\delta (|\xi _i^\mu |^21)+(1a)\delta (\xi _i^\mu )$$
(4)   
with $`a=|\xi _i^\mu |^2`$ the activity of the patterns. Moreover, we assume that there is no bias, i.e., $`\xi _i^\mu =0`$ and that there exist no correlation between patterns such that $`\xi _i^\mu \xi _i^\nu =0`$.

At this point we remark that the long-time behavior of this network model is governed by the spin-1 Hamiltonian

$$H=\underset{i,j}{}J_{ij}\sigma _i\sigma _j\theta _t\underset{i}{}\sigma _i^2.$$
(5)   
Furthermore, the Hopfield model can be recovered by taking the activity $`a=1`$ and the threshold $`\theta _t=0`$.

The standard order parameters of this type of models are the retrieval overlap between the $`\mu `$th-pattern and the microscopic state of the network

$$m_{N,t}^\mu \frac{1}{aN}\underset{i}{}\xi _i^\mu \sigma _{i,t},$$
(6)   
and the neural activity of the neurons

$$q_{N,t}\frac{1}{N}\underset{i}{}|\sigma _{i,t}|^2.$$
(7)   
In the next Section we use these order parameters in order to study the retrieval quality of the network.

## III Mutual information

It is known that the Hamming distance between the state of the network and the pattern $`\{\xi _i^\mu \}`$, viz.

$`d_t^\mu {\displaystyle \frac{1}{N}}{\displaystyle \underset{i}{}}|\xi _i^\mu \sigma _{i,t}|^2=a2am_{N,t}^\mu +q_{N,t}`$ (8)   
is a good measure for the retrieval quality of a network when the patterns are uniformly distributed, i.e., when the neural activity $`a=2/3`$. But for low-activity networks it cannot distinguish between a situation where most of the wrong neurons $`(\sigma _i\xi _i^\mu )`$ are turned off and a situation where these wrong neurons are turned on. This distinction is critical in the low-activity three-state network because the inactive neurons carry less information than the active ones . Therefore the mutual information function $`I(\sigma _{i,t};\xi _{i,t}^\mu )`$ has been introduced 

$$I(\sigma _{i,t};\xi _{i,t}^\mu )=S(\sigma _{i,t})S(\sigma _{i,t}|\xi _{i,t}^\mu )_{\xi _t^\mu }$$
(9)   
where $`\xi _{i,t}^\mu `$ is considered as the input and $`\sigma _{i,t}`$ as the output with $`S(\sigma _{i,t})`$ its entropy and $`S(\sigma _{i,t}|\xi _{i,t}^\mu )`$ its conditional entropy, viz.

$`S(\sigma _{i,t})`$ $`=`$ $`{\displaystyle \underset{\sigma }{}}p(\sigma _{i,t})\mathrm{ln}[p(\sigma _{i,t})]`$ (10)   
$`S(\sigma _{i,t}|\xi _{i,t}^\mu )`$ $`=`$ $`{\displaystyle \underset{\sigma }{}}p(\sigma _{i,t}|\xi _{i,t}^\mu )\mathrm{ln}[p(\sigma _{i,t}|\xi _{i,t}^\mu )].`$ (11)   
Here $`p(\sigma _{i,t})`$ denotes the probability distribution for the neurons at time $`t`$ and $`p(\sigma _{i,t}|\xi _{i,t}^\mu )`$ indicates the conditional probability that the $`i`$-th neuron is in a state $`\sigma _{i,t}`$ at time $`t`$ given that the $`i`$-th site of the stored pattern to be retrieved is $`\xi _{i,t}^\mu `$.

The calculation of the different terms of this mutual information for the model at hand proceeds as follows. As a consequence of the mean-field theory character of our model it is enough to consider the distribution of a single typical neuron so we forget about the index $`i`$ in the sequel. We also do not write the time index $`t`$ and the pattern index $`\mu `$.

The conditional probability that the $`ith`$ neuron is in a state $`\sigma _i`$ at time $`t`$, given that the $`ith`$ site of the pattern being retrieved is $`\xi _i`$ can be obtained as follows. Formally writing $`O=O_{\sigma |\xi }_\xi =_\xi p(\xi )_\sigma p(\sigma |\xi )O`$ for an arbitrary quantity $`O`$ and using the complete knowledge about the system $`\xi =0,\sigma =0,\sigma \xi =am,\xi ^2=a,\sigma ^2=q,\sigma ^2\xi =0,\sigma \xi ^2=0,\sigma ^2\xi ^2=an,1=1`$ we arrive at

$`p(\sigma |\xi )`$ $`=(s_\xi +m\xi \sigma )\delta (\sigma ^21)+(1s_\xi )\delta (\sigma ),`$ (12)   
$`s_\xi `$ $`s{\displaystyle \frac{qn}{1a}}\xi ^2,s{\displaystyle \frac{qan}{1a}}.`$ (13)   
At this point we see from (13) that besides $`m`$ and $`q`$ the following parameter

$$n_{N,t}^\mu \frac{1}{aN}\underset{i}{\overset{N}{}}|\sigma _{i,t}|^2|\xi _i^\mu |^2$$
(14)   
will play an independent role in the mutual information function. This quantity is called the activity-overlap since it determines the overlap between the active neurons, $`|\sigma _{it}|=1`$, and the active parts of the memorized patterns, $`|\xi _i^\mu |=1`$. We remark that it also shows up in the alternative expression of the retrieval quality through the performance $`P_t^\mu =1/N_i\delta _{\xi _{i,t}^\mu ,\sigma _{i,t}}`$ (see ). It does not play any independent role in the time evolution of the network, independent of the architecture considered – diluted, layered or fully-connected.

Next, one can verify that the probability $`p(\sigma |\xi )`$ is consistent with the averages

$`m`$ $`={\displaystyle \frac{1}{a}}\sigma _{\sigma |\xi }\xi _\xi ,`$ (15)   
$`q`$ $`=\sigma ^2_{\sigma |\xi }_\xi ,`$ (16)   
$`n`$ $`={\displaystyle \frac{1}{a}}\sigma ^2_{\sigma |\xi }\xi ^2_\xi .`$ (17)   
These averages are precisely equal in the limit $`N\mathrm{}`$ to the order parameters $`m`$ and $`q`$ in eq. (6)-(7) and to the activity-overlap defined in eq. (14).(The fluctuations around their mean values can be neglected according to the LLN, hence the average over a particular $`i`$-site distribution equals the infinite sum over $`i`$).

Using the probability distribution of the memorized patterns (4) we furthermore obtain

$$p(\sigma )\underset{\xi }{}p(\xi )p(\sigma |\xi )=q\delta (\sigma ^21)+(1q)\delta (\sigma ).$$
(18)   
The expressions for the entropies defined above then become

$`S(\sigma )=q\mathrm{ln}{\displaystyle \frac{q}{2}}(1q)\mathrm{ln}(1q),`$ (19)   
$`S(\sigma |\xi )_\xi =aS_a+(1a)S_{1a},`$ (20)   
$`S_a={\displaystyle \frac{n+m}{2}}\mathrm{ln}{\displaystyle \frac{n+m}{2}}{\displaystyle \frac{nm}{2}}\mathrm{ln}{\displaystyle \frac{nm}{2}}`$ (21)   
$`(1n)\mathrm{ln}(1n),`$ (22)   
$`S_{1a}=s\mathrm{ln}{\displaystyle \frac{s}{2}}(1s)\mathrm{ln}(1s)`$ (23)   
and the mutual information is then given by eq. (9). We recall that $`m_t`$, $`q_t`$ as well as $`n_t`$ are needed in order to completely know the mutual information content of the network at time $`t`$.

## IV Threshold Dynamics

It is known that the parallel dynamics of fully connected networks is difficult to solve, even at zero temperature, because of the strong feedback correlations . Recently, a recursive dynamical scheme has been developed which calculates the distribution of the local field at a general time step using signal-to-noise analysis techniques . Recursion relations are obtained determining the full time evolution of the order parameters. We shortly review these results.

Suppose that the initial configuration of the network $`\{\sigma _{i,0}\}`$, is a collection of IIDRV with mean $`\sigma _{i,0}=0`$, variance $`(\sigma _{i,0})^2=q_{in}`$, and correlated with only one stored pattern, say the first one $`\{\xi _i^1\}`$:

$$\frac{1}{N}\underset{i}{}\xi _i^\mu \sigma _{i,0}=\delta _{\mu ,1}m_{in}^1a,m_{in}^1>0.$$
(24)   
This implies that by the law of large numbers (LLN) one gets for the retrieval overlap and the activity at $`t=0`$

$`m_0^1`$ $``$ $`\underset{N\mathrm{}}{lim}m_{N,0}^1={\displaystyle \frac{1}{a}}\xi _i^1\sigma _{i,0}=m_{in}^1`$ (25)   
$`q_0`$ $``$ $`\underset{N\mathrm{}}{lim}q_{N,0}=\sigma _{i,0}^2=q_{in}.`$ (26)   
At a given time step of the dynamics, the state of a neuron, $`\sigma _{i,t+1}`$ is determined by its local field at the previous time step. In general, in the limit $`N\mathrm{}`$ the distribution of the local field at time $`t+1`$ consists out of a discrete part and a normally distributed part 

$`h_{i,t}=\xi _i^1m_t^1+\sqrt{\alpha aD_t}𝒩(0,1)+B_{i,t}`$ (27)   
$`D_t=\text{Var}\left[r_t^\nu \right]=\text{Var}\left[\underset{N\mathrm{}}{lim}{\displaystyle \frac{1}{a\sqrt{N}}}{\displaystyle \underset{i}{}}\xi _i^\nu \sigma _{i,t}\right],\nu >1`$ (28)   
$`B_{i,t}={\displaystyle \underset{t^{}=0}{\overset{t1}{}}}\alpha \left[{\displaystyle \underset{s=t^{}}{\overset{t1}{}}}\chi _s\right]\sigma _{i,t^{}}`$ (29)   
with $`𝒩(0,1)`$ a Gaussian random variable with mean zero and variance $`1`$ and $`\chi `$ the susceptibility

$$\chi _t=\frac{1}{\sqrt{\alpha aD_t}}𝒟zzF_{\theta _t}\left(\xi ^1m_t^1+\sqrt{\alpha aD_t}z\right)$$
(30)   
where $`𝒟`$ is the Gaussian measure. In the above $``$ denotes the average both over the distribution of the embedded patterns $`\{\xi _i^\mu \}`$ and the initial configurations $`\{\sigma _{i,0}\}`$. The average over the initial configurations is hidden in an average over the local field through the updating rule (1).

The first term on the r.h.s. of (27) is the signal term produced by the pattern that is being retrieved, the rest represents the noise induced by the $`(p1)`$ non-condensed patterns. In particular, the second term is Gaussian noise and the last term $`B_{i,t}`$ contains discrete noise coming from the feedback correlations. The quantity $`D_t`$ satisfies the recursion relation

$$D_{t+1}=\frac{q_{t+1}}{a}+\chi _t^2D_t+2\chi _t\text{Cov}[𝒩(0,q_{t+1}/a),r_t^\mu ]$$
(31)   
For more details we refer to . Using the above scheme the order parameters at a general time step can then be obtained in the limit $`N\mathrm{}`$ from Eqs. (6)-(7) and (1)

$`m_{t+1}^1`$ $`=`$ $`{\displaystyle \frac{1}{a}}\xi _i^1F_{\theta _t}(h_{i,t})`$ (32)   
$`q_{t+1}`$ $`=`$ $`F_{\theta _t}^2(h_{i,t}).`$ (33)   
The activity overlap needed in order to find the mutual information can also be written as

$$n_{t+1}^1=\frac{1}{a}(\xi _i^1)^2F_{\theta _t}^2(h_{i,t}).$$
(34)   
Of course, we then also need to specify its initial value

$$n_0^1\underset{N\mathrm{}}{lim}n_{N,0}^1=\frac{1}{a}(\xi _i^1)^2(\sigma _{i,0})^2.$$
(35)   
The idea of the self-control threshold dynamics introduced in the diluted model  and studied for some other models without feedback correlations  has been precisely to let the network counter the noise term in the local field at each step of the dynamics by introducing the following form

$$\theta _t=c(a)\sqrt{\alpha aD_t},$$
(36)   
where the function $`c(a)`$ is a function of the pattern activity. We remark that in these cases without feedback there is no discrete noise in the local field (the term $`B_{i,t}`$ in eq. (27) is absent). Furthermore, also the covariance term in eq. (31) is absent. Moreover, for the diluted model $`D_t=q_t/a`$, i.e., only the first term in eq. (31) is present. So this dynamical threshold has two important characteristics. First, it is a macroscopic parameter having the same value for every neuron thus no average must be taken over the microscopic random variables at each time step. Secondly, it changes each time step but no statistical history intervenes in this proces.

We see that the choice (36) is in fact related to the variance of the local field, taken for a fixed realization of the pattern which is being retrieved. It is the width of the noise produced by the non-condensed patterns. It is obvious that it cannot be taken to be a function of the overlap with the pattern being retrieved. As a consequence, for the fully connected network we cannot work with the exact form for $`D_t`$ as given in eq. (31) because of the presence of the covariance. So, if we want to take into account some effects of feedback correlations and if we want the threshold to have the characteristic properties mentioned above, we need to approximate the covariance term in eq. (31) such that only the previous time step is involved. This is realised by approximating this term by $`2\chi _t\{\text{Var}[𝒩(0,q_{t+1}/a)]\text{Var}[r_t^\mu ]\}^{1/2}=2\chi _t[q_{t+1}/a]^{1/2}[D_t]^{1/2}`$. We then easily get

$`\alpha aD_{t+1}`$ $`=`$ $`\left[G_t+\sqrt{\alpha q_{t+1}}\right]^2`$ (37)   
$`G_t`$ $`=`$ $`{\displaystyle 𝒟zzF_{\theta _t}\left(\xi ^1m_t^1+\sqrt{\alpha aD_t}z\right)}`$ (38)   
Furthermore, we take both contributions at equal times and call $`\sqrt{\alpha aD_t}\mathrm{\Delta }_t=G_t+\sqrt{\alpha q_t}`$. For more details on this approximation of the feedback correlations we refer to  and references therein. Finally, since $`G_t`$ is a function of the overlap $`m_t^1`$, a quantity which is not available to the network we replace it by $`G_0=\sqrt{2/\pi }a`$.

What is left then is to find a form for $`c(a)`$. For the low-activity networks considered up to now the storage capacity could be considerably improved by taking $`c(a)=[2\mathrm{ln}(a)]^{1/2}`$ such that for the diluted model $`\theta _t=[2\mathrm{ln}(a)\alpha q_t]^{1/2}`$. The same form has been shown to work for the layered model . For the fully connected model considered here we again propose, a priori, this form. So, combining these results we take as self-control threshold

$`\theta _t`$ $`=`$ $`\sqrt{2\mathrm{ln}(a)}\mathrm{\Delta }_t^0`$ (39)   
$`\mathrm{\Delta }_t^0`$ $`=`$ $`\sqrt{2/\pi }a+\sqrt{\alpha q_t}.`$ (40)   
Finally, we make one more assumption on the dynamics. In the local field distribution (27) we forget about the discrete noise $`B_{i,t}`$ and suppose that the noise produced by the non-condensed patterns is Gaussian distributed. Computer simulations have shown that this assumption is approximately valid as long as the retrieval is succesful . As a consequence we can write down recursion relations for the order parameters

$`m_{t+1}`$ $`=`$ $`{\displaystyle 𝒟zF_{\theta _t}(m_t^1+z\mathrm{\Delta }_t)}`$ (41)   
$`q_{t+1}`$ $`=`$ $`a{\displaystyle 𝒟z[F_{\theta _t}(m_t^1+z\mathrm{\Delta }_t)]^2}`$ (43)   
$`+(1a){\displaystyle 𝒟z[F_{\theta _t}(z\mathrm{\Delta }_t)]^2}`$   
with

$`\mathrm{\Delta }_t`$ $`=`$ $`\sqrt{\alpha q_t}+a{\displaystyle 𝒟zzF_{\theta _t}(m_t^1+z\mathrm{\Delta }_t)}`$ (45)   
$`+(1a){\displaystyle 𝒟zzF_{\theta _t}(z\mathrm{\Delta }_t)}`$   
where we have already averaged over $`\xi `$. We remark that the form of the last equation is different from the corresponding equation for the diluted and the layered versions of this model  because of the feedback.

The expressions for the overlap $`m_{t+1}`$, the neural activity $`q_{t+1}`$ and the noise $`\mathrm{\Delta }_t`$ due to the non-condensed patterns describe the (approximate) macro-dynamics of the fully-connected neural network. Besides the self-control model with the threshold given by eq. (39)-(40) we also consider the model with the threshold fixed at its zero time value, i.e., $`\theta _t=\theta _0`$.

At this point we remark that when studying the mutual information, we want to introduce explicitly the activity-overlap parameter, $`n_t`$ (recall eq. (14)) in the dynamics leading to the following expression for $`q_{t+1}`$

$$q_{t+1}=an_{t+1}+(1a)s_{t+1}$$
(46)   
where $`s_{t+1}`$ is then, obviously, defined by the integral of $`[F_{\theta _t}(z\mathrm{\Delta }_t)]^2`$. This parameter $`s`$ is precisely that introduced in Eq. (13) and measures the number of active neurons in inactive condensed pattern sites.

In the following section we compare the retrieval properties of a fully connected network governed by this approximate dynamics with and without self-control with numerical simulations. The main aim is to show that self-control also works in the case of fully connected models.

## V Results

### A Numerical Results

We have solved the time evolution of our threshold dynamics with a time-dependent self-control threshold given by eq. (39)-(40)) and with a time-independent threshold where the neural activity is fixed by $`q_t=q_0`$.

We have studied the behavior of these networks in the range of pattern activities $`0.01a0.67`$ i.e., from low-activities to a uniform distribution of patterns.

For both thresholds it turned out that the best results were obtained by taking $`c(a)=\sqrt{2\mathrm{ln}(a)}+K`$, with $`K=0`$ for $`a0.1`$ and $`K=0.5`$ for $`a<0.1`$. At this point we remark, however, that the pure log form for $`c(a)`$ is derived in the theoretical limit $`a0`$. So, it may be that we did not reach small enough values in our numerical analysis (which is due to numerical complexity). We recall that one of the main aims of this work is to show that self-control also works for fully connected models.

The important features of self-control are illustrated in Figs. 1-5. In Fig. 1 we compare the time evolution of the retrieval overlap, $`m_t`$, starting from several initial values, $`m_0`$, for the model with self-control, $`\theta _{sc}=\theta _t`$ (recall eq. (39)-(40)) with the model with fixed threshold $`\theta _0`$. An initial neural activity $`q_0=a=0.01`$ and a loading $`\alpha =2`$ have been taken. We observe that the self-control forces more of the overlap trajectories to go to the retrieval attractor $`m=1`$. Only an initial overlap $`m_00.4`$ for the self-control model versus $`m_00.6`$ for the fixed threshold model is needed. We remark that for $`m_00.6`$ the overlap decreases in the first time step for both models. This is an expression of the fact that correlations are especially important in the first time steps leading to a decreasing neural activity $`q_t`$, but the self-control threshold is able to counter these effects. Near the attractor correlations seem to become less important and the Gaussian character of the local field distribution dominates.

Since the initial overlap needed to retrieve a pattern is smaller for the self-control model, the basins of attraction of the patterns are substantially larger. This is further illustrated in Figs. 2-4 where the basin of atraction for the whole retrieval phase $`R`$ is shown for both models with an initial value $`q_0=a=0.01`$. We have calculated the fixed-point $`m_{\mathrm{}}`$ of the dynamics (41), (43) and (45) and we have determined the intial conditions of the relevant parameters such that the network is able to retrieve, i.e., such that $`m_{\mathrm{}}1`$. It is interesting to also give $`n_0`$ and/or $`s_0`$ separately in order to see how the activity $`q_t`$ is built up.

In Fig. 2 we have used $`q_0=a,n_0=1`$. The basin of attraction for the self-control model is larger, even near the border of critical storage. Hence the storage capacity itself is also bigger.

Furthermore, a smaller initial activity-overlap $`n_0`$ suffices to have retrieval as is seen in Fig. 3. There we start with initial conditions $`m_0=0.4`$, i.e., the smallest initial overlap possible for $`\alpha =2`$ as we recall from Fig. 1, and $`q_0=an_0`$ or, equivalently $`s_0=0`$. So we consider small $`q_0`$ running from $`0.004`$ to $`0.01`$. We observe the peculiar behavior that for the fixed-threshold network an initial $`n_0>m_0`$ is needed, but even then still no retrieval is possible for low storage $`\alpha <0.1`$. For the self-control model a much broader region of retrieval exists. Finally, the specific role of the parameter $`s_t`$ is displayed in Fig. 4. We start from a maximal initial overlap $`m_0=1`$ and take $`n_0=1`$ meaning that for $`s_0=0`$ to $`s_0=1`$, $`q_t`$ runs from $`0.01`$ to $`1`$. It can be seen that especially when $`s_0`$ is getting large the storage capacity of both models decreases quite drastically but again much less for the self-control model.

We conclude with the observation that self-control works in a large range of pattern activities $`a`$, as shown in Fig. 5. There the mutual information content $`i=_i_\mu I/(\mathrm{\#}J_{ij})=\alpha I`$ is plotted as a function of the loading $`\alpha `$ on a logaritmic scale. We observe the slow increase of $`i`$ as the activity $`a`$ decreases, saturating at a value close to $`i0.3`$. This behavior is typical for low activity networks .

### B Simulations

Simulations have been carried out for systems with $`N=10^4`$ neurons. For every new stored pattern $`\mu `$, we start our dynamics with a state $`\sigma _i=\xi _i^\mu `$, and calculate the order parameters $`m_t`$, $`n_t`$ and the activity overlap $`q_t`$, using the definitions (6), (7) and (14). To avoid a very large computation time we have stopped the dynamics after $`t5`$ time steps when no convergence was reached before. Then we have averaged over windows in the $`p`$-axis in order to obtain the mutual information $`i`$. The window size runs from $`\delta p=50`$ for $`a=0.67`$ (where we have stored $`p=10^3`$ patterns) up to $`\delta p=2\times 10^3`$ for $`a=0.01`$ (where we have stored $`p=5\times 10^4`$ patterns).

The conditions on the LLN mentioned in Section IVA are approximatelly fullfilled for such large networks, since the fluctuations (neglected in Eqs.(15)-(17)) are of order $`1/\sqrt{aN}`$. However, for smaller activities $`a`$, this quantity may be not so small. It becomes crucial in the case $`a=0.01`$, where this quantity is $`0.1`$, such that the finite size effects get relevant. This implies a kind of cut-off in the information for the self-control model as seen in Fig. 6. However, the agreement with the analytic results of Fig. 5 is quite good up to $`a=0.01`$.

In order to further understand the details of the retrieval quality we plot in Fig. 7 all the parameters $`m,n,q`$ for the model with and without self-control in two cases: $`a=0.01`$, a low-activity case, and $`a=0.67`$, implying a uniform distribution of patterns. For the uniform case we do not see a big difference between the two models (self-control and fixed threshold). Only for larger values of $`\alpha `$, self-control shows a little improvement. For the low-activity case, however, the main role of self-control on the neural activity is clearly noticed since $`qa`$ in that case while in the fixed-threshold model it is impossible to control $`q`$ such that it stays in the neigborhood of $`a`$. As a conseqeunce the mutual information, e.g., is only about half of that for the model with self-control.

Finally, in Fig. 8 we compare the simulations with the results from the fixed-points of the dynamics (41), (43) and (45) for $`a=0.03`$. Up to $`t=5`$ time steps are considered for both the model with and without self-control and we have averaged over a window in the $`p`$-axis of size $`\delta p=10^3`$. For the self-control model the small underestimation of the theoretical results can, of course, be attributed to the approximations of the noise term (recall Eqs (37) and (40)).

## VI Concluding Remarks

In this paper we have introduced a self-control threshold in the dynamics of fully connected networks with three-state neurons. This leads to a large improvement of the quality of retrieval of the network. The relevant quantity in order to study this, especially in the limit of low activity is the mutual information function. The mutual information content of the network as well as the critical capacity and the basins of attraction of the retrieval solutions for three-state patterns are shown to be larger because of the self-control mechanism. Furthermore, since the mutual information saturates, the critical capacity of the low-activity network behaves as $`\alpha _c=O(|a\mathrm{ln}(a)|^1)`$. Numerical simulations confirm these results.

This idea of self-control might be relevant for various dynamical systems, e.g., when trying to enlarge the basins of attraction and convergence times. Indeed, it has been shown to work also for both diluted and layered networks. Binary as well as ternary neurons and patterns have been treated. In all cases, it turns out that in the low-activity regime the self-control threshold can be taken to be proportional to the square root of the neural activity of the network.

## Acknowledgments

We would like to thank G. Jongen for useful discussions. This work has been supported by the Research Fund of the K.U.Leuven (grant OT/94/9). One of us (D.B.) is indebted to the Fund for Scientific Research - Flanders (Belgium) for financial support.