{'fraction_non_alphanumeric': 0.034161771351107095, 'fraction_numerical': 0.012245820153637597, 'mean_word_length': 4.207294117647058, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On-line learning and generalisation in coupled perceptrons

We study supervised learning and generalisation in coupled perceptrons trained on-line using two learning scenarios. In the first scenario the teacher and the student are independent networks and both are represented by an Ashkin-Teller perceptron. In the second scenario the student and the teacher are simple perceptrons but are coupled by an Ashkin-Teller type four-neuron interaction term. Expressions for the generalisation error and the learning curves are derived for various learning algorithms. The analytic results find excellent confirmation in numerical simulations.


Introduction
One of the more interesting properties of neural networks is their ability to learn from examples. In on-line learning processes a student network updates its couplings after the presentation of each example in order to make its outputs agree with the outputs of the teacher. In the standard situation the student knows only the inputs and the corresponding outputs of the teacher and has no further knowledge of the rule used by the latter. Furthermore, in the course of learning the student is able to classify correctly also new examples, which it has never seen before. The latter property is called generalisation.
Various aspects of learning and generalisation in neural networks have been intensively studied in many different contexts. For about a decade now statistical mechanical methods have been used successfully in these studies (for recent reviews see, for example [1,2,3,4]).
A lot of the theoretical research has been concentrated on the simplest models, such as the binary perceptron. Parallel to the progress in these investigations, new more realistic models have been considered, e.g., models with multi-state neurons [5], models with multi-neuron interactions [6,7], models with many layers (see, e.g, [8,9,10]).
In this paper we study on-line learning and generalisation in a recently introduced model, allowing two different types of binary neurons at each site, possibly having different functions [11,12]. More specifically, this so called Ashkin-Teller (AT) perceptron contains, besides two-neuron interaction terms, also a four-neuron interaction term. For the underlying biological motivation for the introduction of different types of neurons we refer to [13]. Here, we recall that the maximal capacity of the AT perceptron model II introduced in [11,12] can be larger than the one of the standard binary perceptron [12] and that the corresponding recurrent network model can be a more efficient associative memory than a sum of two Hopfield models [13]. A natural question is then how this AT perceptron performs in on-line learning and generalisation tasks.
Two learning scenarios turn out to be of interest. In the first scenario where the student and the teacher are independent AT perceptrons, we show that the resulting learning curves do not differ very much from the already known ones for perceptrons with multi-state neurons. For some particular values of the network parameters we precisely reproduce the learning curve of the 4-state Potts perceptron [5].
In the second scenario both the student and the teacher are represented by a simple perceptron but they are coupled by an AT type four-neuron interaction term. Hence, contrary to the standard setup, they are not independent. This can be considered as a sort of "hardware" coupling. As a result, also the teacher mapping is changing in the process of learning. We obtain a set of learning curves which qualitatively differ from those found in the independent setup. We also find different asymptotic behaviour when the number of examples increases to infinity. For certain values of the network parameters such a coupling describes the realistic situation that the rule used by the teacher is partially shared by the student.
The rest of the paper is organised as follows. In section 2 the model and the learning scenarios are introduced. The formulas for the generalisation error are derived in section 3. The differential equations for the evolution of the order parameters are obtained in section 4. Their solutions, compared with numerical simulations can be found in section 5. In section 6 some concluding remarks are presented. Finally, two appendices contain some technical details of the derivations.

The model and the learning scenarios
The AT perceptron is defined as a mapping of the binary (±1) inputs {s i , σ i }, i = 1, ..., N into two binary (±1) outputs s and σ: where θ is the Heaviside step function and γ r ≥ 0, r = 1, 2, 3, denote the strength of the local fields h r which are defined as follows The mapping (1)-(2) can be equivalently represented by the set of three equations (cfr. model I in [12]) For γ 3 = 0 the outputs s and σ are completely independent and defined like in the simple perceptron

Learning scenario I
First, we consider the standard situation where the student and the teacher are two completely independent networks. In our case they are represented by AT perceptrons meaning that the outputs of the teacher {s T , σ T } and of the student {s S , σ S } are both determined by the mapping (1)-(2) but with different couplings: J T r and J S r respectively, with J r = {J (r) i }. Initially, the student and the teacher couplings are not correlated. At each time step t an example is presented to the student. The student network then updates its couplings according to the following learning rule F where In this scenario we consider only Hebbian learning for which F = 1. Furthermore, examples are chosen randomly with equal probability out of the complete set of examples.

Learning scenario II
Alternatively, the AT perceptron can also be seen as two coupled perceptrons, with outputs s and σ. In the second scenario we precisely analyse learning between such coupled perceptrons (or branches of the AT perceptron). The outputs of the student s and the teacher σ are defined by the equations (1) and (2) respectively. When h 3 > 0, the teacher and the student use two different mixtures of two perceptron mappings defined by the couplings J 1 and J 2 . It implies that s and σ are always equal to sgn(h 1 ) or sgn(h 2 ) and sometimes, depending on the relation between γ 1 h 1 , γ 2 h 2 and γ 3 h 3 , s = σ. In the limit γ 3 → ∞ the student and the teacher network become so strongly coupled that one always has s = σ and the mapping (1)-(2) can be simplified to For h 3 < 0, the situation is quite different. Even with J 1 = J 2 there is always a non-zero fraction of disagreements between the student and the teacher, as long as γ 3 > 0. In the limit γ 3 → ∞ the student always disagrees with the teacher, and the mapping (1)- (2) can be written in the form: For any value of the coupling field h 3 and γ 3 = 0 the student and the teacher are independent and they use the mappings defined by only one coupling vector (cfr. (7)-(8)).
In the sequel we take s = σ because the student and the teacher must have the same inputs. We remark that this implies that h 3 = i J i /n 3 (cfr. (3)). Again, at each time step t an example is presented to the student network and its coupling vector J 1 is updated as follows Furthermore, at each time step a new coupling vector J 3 is generated thus making the coupling between the perceptrons random. The coupling vector of the teacher, J 2 , is not changed in the process of learning, but later on we average over all possible teachers.
In this scenario we consider three learning rules F :

Generalisation error
A quantity of interest in the sequel is the generalisation error. It is defined as the probability that the student and the teacher disagree, i.e. that their outputs are different. When the teacher and the student are simple independent perceptrons the generalisation error ε g = arccos(ρ)/π is a simple function of the overlap ρ = J T · J S /(n S n T ) between the student and the teacher couplings, which in this case plays the role of an order parameter. Unfortunately, for more complicated models this relation takes a much more involved form (see, e.g., [5]).

Scenario I
In the first scenario the definition of the generalisation error reads with the overlaps ρ r defined by and with . . . I = dh T dh S . . . P I (h T , h S ) denoting the average over the teacher field, The averages over these fields are double averages, one over the examples and one over the couplings. This arises because the couplings and the examples enter the mapping (1)-(2) and the learning rules only through the local fields. We assume that the examples are taken randomly with equal probability out of the full training set. Then, in the thermodynamic limit the local fields become correlated Gaussian variables and the joint probability distribution P I (h T , h S ) can be written down in the form Performing the averages in (16) explicitly leads to the expression where is a correlated Gaussian.

Scenario II
In the second scenario the generalisation error is given by with the overlap ρ defined by Here again, as in the first scenario, the average over the examples and the couplings is done through averaging over the local fields. The examples are chosen randomly with equal probability out of the full set of examples. In the thermodynamic limit this leads to a Gaussian distribution of the local fields. Since the behaviour of the system strongly depends on the sign of the coupling field h 3 we consider three different field distributions P II In the case of the distribution P ± the components of the vector J 3 are taken randomly (with equal probability) from some interval (−a, a), with a a positive real number. In the case of the distributions P + and P − these components are chosen in the same way but those values which lead to negative respectively positive values of the field h 3 are omitted. The generalisation error in these three situations reads, with obvious notation ε c g (ρ) = 1 π arccos(ρ) + I c c = ±, +, − where and It is easy to realize that only for positive h 3 (i.e. for P II = P + ) the generalisation error ε + g (ρ) goes to zero as ρ goes to 1. It is also equal to zero for any ρ when P II = P + and γ 3 = ∞.

Order parameters and their evolution
As can be seen from the formulas written down in the last section, the generalisation error is a function of the overlaps ρ or ρ r , which play the role of order parameters in the learning process. Their evolution is coupled with the evolution of the norms of the couplings n r and in the thermodynamic limit N → ∞ it can be described by ordinary differential equations [14].
In the first scenario a standard calculation (for a review see, e.g., [2]) leads to the following result for Hebbian learning d dα where Ψ T 1 = s T , Ψ T 2 = σ T , Ψ T 3 = s T σ T and α = t/N is the number of examples scaled with the size of the system. It becomes continuous in the thermodynamic limit. After performing the averages we arrive at with the quantity b r given by For γ 1 = γ 2 = γ 3 this quantity simplifies to We remark that the differential equations (32) for a given r have the same form as the differential equations found for the simple perceptron with Hebbian learning [2]. More specifically, they differ only by the value of the coefficient b r , which for the simple perceptron is equal to 2/π ∼ = 0.798. For the Hebbian learning we are considering, it is possible to construct a simple expression for ρ r as a function of α. Following Opper and Kinzel [1] we slightly modify the update rule (9), (10), (11) (substituting 1/N by 1/ √ N ) and easily arrive at where we have taken as initial condition ρ(0) = 0 and where This expression differs from the solution of (32) only for small values of α and has the advantage of having a simple form. The evolution of ρ in the case of simple perceptrons is described by the single equation (36), but with a coefficient a r = √ 2. Since these results are very similar to the results obtained for the simple perceptron we do not test other algorithms in this scenario because we expect that also in those cases a strong resemblance to the simple perceptron occurs.
In the second scenario with the learning rule F defined in subsection 2.2 we have to solve the following set of differential equations: d dα Performing the averages leads to much more complicated expressions than those obtained in the first scenario. The explicit form of these expressions obtained for Hebbian, perceptron and Adatron learning with the distributions P ± and P + can be found in Appendix A.

Results
In this section we discuss the numerical solutions of the differential equations (31) and (38)-(39) and compare them with the results of simulations. Because only the ratios of the strength parameters γ 1 , γ 2 and γ 3 are important we take γ 1 = γ 2 = 1, and vary only γ 3 .

Scenario I
The learning curves for small values of the number of examples α obtained in the first scenario using formula (36) are presented in figure 1. All curves start with an initial generalisation error ε g = 0.75 corresponding to random guessing in four-state models.
For γ 3 = 0 learning between two independent perceptrons is described. For γ 3 = 1 the learning curve is identical with the one of the 4-state Potts perceptron [5] (cfr. [11,12]). In the limit α → ∞, ε g decays like α − 1 2 for all values of γ 3 , precisely like in the case of learning between simple perceptrons.

Scenario II
A careful analysis of expression (28) leads to the conclusion that in the second scenario the generalisation error can be nonzero even when the normalised angle between the student and the teacher couplings, φ = arccos(ρ)/π, is equal to zero. This happens when we allow the field h 3 to take negative values. Therefore, we follow the evolution of two dynamical variables in the sequel: the generalisation error ε g and the normalised angle between the student and the teacher φ. For all the learning algorithms and distributions of the fields that we have considered, we observe an abrupt change in the asymptotic behaviour in α when γ 3 changes from 0 to some non-zero value. Logarithmic plots of the learning curves for two distributions of the fields, P ± and P + , are presented in figures 2-7. The learning curves for the distribution P − are qualitatively very similar to the curves obtained for P ± . 5.2.1. P II = P ± Let us first analyse the results obtained for the distribution P ± in more detail. For γ 3 = 0, the generalisation error saturates at some non-zero value. For Hebbian and perceptron learning the angle φ between the student and the teacher is asymptotically decreasing to zero at a higher rate than in the decoupled case γ 3 = 0. For Hebbian learning we find that in the limit α → ∞, φ ∼ α −1 , versus φ ∼ α − 1 2 for γ 3 = 0, while in the case of the perceptron algorithm φ ∼ α − 1 2 , versus φ ∼ α − 1 3 for γ 3 = 0. For the Adatron algorithm φ and ε g both saturate at some non-zero value. In spite of the fact that the generalisation error never vanishes the student is able to learn the couplings of the teacher using the Hebbian or perceptron algorithm. 5.2.2. P II = P + We observe that for all algorithms the generalisation error goes asymptotically to zero. For Hebbian and perceptron learning it decreases faster than in the decoupled case. In the limit α → ∞, we get ε g ∼ α −1 for Hebbian learning while ε g ∼ α − 1 2 for perceptron learning. For Adatron learning we obtain the same decay exponent as in the decoupled case. Surprisingly, for the perceptron and Adatron algorithms the decay of the angle between the student and the teacher, φ, is slower than in the decoupled case in the limit α → ∞. For the perceptron we have φ ∼ α − 1 4 and for the Adatron we find φ ∼ α − 1 2 . On the contrary, for Hebbian learning φ ∼ α − 1 2 as for the decoupled case.
Since an analytic analysis of the differential equations (see Appendix A) is rather involved, the asymptotic exponents discussed above have been determined numerically. Only in the case of Hebbian learning with the field distribution P + the numerical analysis was not entirely unambiguous. Therefore, we have derived the corresponding exponents analytically. Details can be found in Appendix B.
The initial generalisation error is a function of the strength parameter γ 3 , which measures the strength of the coupling between the two perceptrons. The larger the γ 3 the bigger the common knowledge between the student and the teacher, so the smaller the initial error. For γ 3 → ∞, the student and the teacher use precisely the same rule (13) in order to determine their outputs.
Finally, the numerical solution of the equations (38)-(39) suggests that there is a simple relation between the decay exponents of φ and ε g , denoted by y φ and y g respectively, This relation can also be derived analytically (see Appendix B). For γ 1 = γ 2 we find in the limit α → ∞ (and φ → 0) that confirming the observation (40).

Computer simulations
To check the analytic results described above we have performed numerical simulations. The system sizes have been varied between N = 100 and N = 999 neurons. An excellent agreement has been found for both scenarios and all learning algorithms, even for relatively small N. As a representative example we present a comparison between simulations and analytic results obtained in the second scenario with the Adatron algorithm for γ 3 = 0.1 and P II = P ± . For the sake of clarity we show the results obtained for small and big α separately. The analytic results for small α are compared with simulations for a system with N = 999 neurons ( fig. 8). For bigger α we have made simulations for smaller systems (N = 100), which are displayed in fig. 9. In both cases only the results obtained for one sample are shown. For small α the simulations are smoothly aligned along the theoretical curves. This points to the self averaging property of the learning process. For bigger values of α very strong fluctuations occur around the theoretical result. This happens only for the Adatron algorithm and P II = P ± and, hence, cannot be explained entirely by the relatively small size of the system. Indeed, as has been noticed in section 5, in this case there is always a non-zero fraction of disagreement between the student and the teacher. So, a strategy used by the Adatron algorithm which updates the couplings proportional to the error made by the student, must lead to rather big random changes. Nevertheless the simulation points in fig. 9 are evenly distributed on both sides of the theoretical curve.

Conclusions
In this paper we have studied on-line learning and generalisation using the AT perceptron. Two learning scenarios have been considered. The results obtained in the first scenario, where the student and the teacher are represented by independent AT perceptrons, are very similar to the results obtained for the simpler models [2]. For a particular choice of the network parameters the learning curve precisely reproduces that found for the 4-state Potts perceptron [5].
In the second scenario the student and the teacher are taken to be simple perceptrons coupled by a four-neuron interaction term. Particular results depend crucially on the distribution of the couplings J 3 .
For the field distribution P II = P ± the generalisation error always saturates at some non-zero value. This is not surprising since this distribution allows the field h 3 to take negative values what inevitably leads to a non-vanishing fraction of disagreements between the student and the teacher even when J 1 = J 2 (cfr. (1), (2)). In spite of this, for Hebbian and perceptron learning the student manages to learn the couplings of the teacher perfectly (in the limit α → ∞). This does not happen, however, for the Adatron algorithm, which in the standard (decoupled) situation proved to be the fastest [2]. The reason is that this algorithm changes the couplings of the student proportionally to the error made by the latter. Since this error is non-zero even for J 1 = J 2 , this cannot be a good strategy. Hence, the more "blind" updates (Hebbian and perceptron) appear to be more effective.
For P II = P + we have obtained quite different results. In this case the generalisation error goes to zero when ρ goes to 1. For Hebbian and perceptron learning we observe faster decay of ε g than in the decoupled case. For Adatron learning the decay exponent of ε g is the same as for γ 3 = 0. Surprisingly, for all algorithms we find the same or slower decay of φ compared with the decoupled case.
The best asymptotic decay of the generalisation error has been obtained for P II = P + with the Adatron rule: ε g ∼ 0.618α −1 . Comparing with the case of independent perceptrons we see that it is better than the lower bound for on-line learning [2] (ε g ∼ 0.88α −1 ) and worse than the Bayesian lower bound [15] (ε g ∼ 0.44α −1 ).
We remark that in the course of a learning process in the second scenario also the teacher mapping is changed but not the teacher couplings. This can be interpreted as a kind of effective mutual learning caused by the ("hardware") coupling of the two perceptrons. This is different from the mutual learning process analysed in [16,17], the only other learning process of this type known to us. There, in contrast to our setup, the teacher explicitly learns from the student. In our model the decay exponent of ε g is not influenced by a particular value of the strength parameter γ 3 as long as it is nonzero.
The model analysed in the second scenario with P II = P + where a part of the learning rule is shared by the teacher and the student, can be compared to a real life situation in which both of them, e.g., have the same cultural background, followed the same education . . . One can expect that in such a situation the learning process is much more efficient since the student and the teacher speak in a sense the same language. It corresponds to a faster asymptotic decay of the generalisation error in our model. It would be interesting to see, e.g., whether an optimisation of the learning process [14] would still improve these results.