{'fraction_non_alphanumeric': 0.05080757726819542, 'fraction_numerical': 0.008893320039880359, 'mean_word_length': 4.236166214240969, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Some Theoretical Properties of a Network of Discretely Firing Neurons

The problem of optimising a network of discretely firing neurons is addressed. An objective function is introduced which measures the average number of bits that are needed for the network to encode its state. When this is minimised, it is shown that this leads to a number of results, such as topographic mappings, piecewise linear dependence on the input of the probability of a neuron firing, and factorial encoder networks.


Introduction
In this paper the problem of optimising the firing characteristics of a network of discretely firing neurons will be considered. The approach adopted will not be based on any particular model of how real neurons operate, but will focus on theoretically analysing some of the information processing capabilities of a layered network of units (which happen to be called neurons). Ideal network behaviour is derived by choosing the ideal neural properties that minimise an information theoretic objective function which specifies the number of bits required by the network to encode the state of its layers. This is done in preference to assuming a highly specific neural behaviour at the outset, followed by optimisation of a few remaining parameters such as weight and bias values.
Why use an objective function in the first place? An objective function is a very convenient starting point (a set of "axioms", as it were), from which everything else can, in principle, be derived (as "theorems", as it were). An objective function has the same status as a model, which may be falsified should some counterevidence be discovered. The objective function used in this paper is the simplest that is consistent with predicting a number of non-trivial results, such as topographic mappings, and factorial encoders (which are discussed in this paper). However, it does not include any temporal information, nor any biological plausibility constraints (other than the fact that the network is assumed to be layered). More complicated objective functions will be the subject of future publications.
In section 2 an objective function is introduced, and its connection with discretely firing neural networks is derived. In section 3 some examples are presented which show how this theory of discretely firing neural networks leads to some non-trivial results.

Theory
In this section a theory of discretely firing neural networks is developed. Section 2.1 introduces the objective function for optimising an encoder, and section 2.2 shows how this can be applied to the problem of optimising a discretely firing neural network.

Objective Function for Optimal Coding
The inspiration for the approach that is used here is the minimum description length (MDL) method [5]. In this paper, a training set vector (which is unlabelled) will be denoted as x, a vector of statistics which are stochastically derived from x will be denoted as y, and their joint probability density function (PDF) will be denoted as Pr(x, y). The problem is to learn the functional form of Pr(x, y), so that vectors (x, y) sampled from Pr(x, y) can be encoded using the minimum number of bits on average. It is unconventional to consider the problem of encoding (x, y), rather than x alone, but it turns out that this leads to many useful results.
Thus Pr(x, y) is approximated by a learnt model Q(x, y), in which case the average number of bits required to encode an (x, y) sampled from the PDF Pr(x, y) is given by the objective function D, which is defined as Now split D into two contributions by using Pr(x, y) = Pr(x) Pr(y|x) and Q(x, y) = Q(x) Q(y|x).
The first term is the cost (i.e. the average number of bits), averaged over all possible values of y, of encoding an x sampled from Pr(x|y) using the model Q(x|y). This interpretation uses that Pr(x) Pr(y|x) = Pr(y) Pr(x|y). The second term is the cost of encoding a y sampled from Pr(y) using the model Q(y). Together these two terms correspond to encoding y (the second term), then encoding x given that y is known. The model Q(x, y) may be optimised so that it minimises D, and thus leads to the minimum cost of encoding (x, y) sampled from Pr(x, y). Ideally Q(x, y) = Pr(x, y), but in practice this is not possible because insufficient information is available to determine Pr(x, y) exactly (i.e. the training set does not contain an infinite number of (x, y) vectors). It is therefore necessary to introduce a parametric model Q(x, y), and to choose the values of the parameters so that D is minimised. If the number of parameters is small enough, and the training set is large enough, then the parameter values can be accurately determined.
A further simplification may be made if y can occupy much fewer states than x (given y) can, because then the cost of encoding y is much less than the cost of encoding x (given y) (i.e. the second and first terms in equation 2, respectively). In this case, it is a good approximation to retain only the first term in equation 2. This approximation becomes exact if Q(y) assigns equal probability to all states y, because then the third term is a constant. The reason for defining the objective function D as in equation 1, rather than defining it to be the first term of equation 2, is because equation 1 may be readily generalised to more complex systems, such as (x, y, z) in which Pr(x, y, z) = Pr(x) Pr(y|x) Pr(z|y), and so on. An example of this is given in section 3.1.
It is possible to relate the minimisation of D to the maximisation of the mutual information I between x and y. If the cost of encoding an x sampled from Pr(x) using the model Q(x) (i.e. −´dx Pr(x) log Q(x)) and the cost of encoding a y sampled from Pr(y) using the model Q(y) (i.e. − y Pr(y) log Q(y)) are both subtracted from D, then the result is −´dx y Pr(x, y) log Q(x,y) Q(x) Q(y) . When Q(x, y) −→ Pr(x, y) this reduces to (minus) the mutual information I between x and y. Thus, if the cost of encoding the correlations between x and y is much greater than the cost of separately encoding x and y (i.e. the log (Q(x) Q(y)) term can be ignored in I), then D-minimisation approximates I-maximisation, which is another commonly used objective function.

Application to Neural Networks
In order to apply the above coding theory results to a 2-layer discretely firing neural network, it is necessary to interpret x as a pattern of activity in the input layer, and y as the vector of locations in the output layer of a finite number of firing events. The objective function D is then the cost of using the model Q(x, y) of the network behaviour to encode the state (x, y) of the neural network (i.e. the input pattern and the location of the firing events), which is sampled from the Pr(x, y) that describes the true network behaviour. For instance, a second neural network can be used solely for computing the model Q(x, y), which is then used to encode the state (x, y) of the above first neural network. Note that no temporal information is included in this analysis, so the input and output of the network is a static (x, y) vector containing no time variables.
These two neural networks can be combined into a single hybrid network, in which the machinery for computing the model Q(x, y) is interleaved with the neural network, whose true behavior is described by Pr(x, y). The notation of equation 2 can now be expressed in more neural terms, where Pr(y|x) is then a recognition model (i.e. bottom-up) and Q(x|y) is then a generative model (i.e. top-down), both of which live inside the same neural network. This is an unsupervised neural network, because it is trained with examples of only x-vectors, and the network uses its Pr(y|x) to stochastically generate a y from each x.
Now introduce a Gaussian parametric model Q(x|y) where x ′ (y) is the centroid of the Gaussian (given y), σ is the standard deviation of the Gaussian. Also define a soft vector quantiser (VQ) objective function D VQ as which is (twice) the average Euclidean reconstruction error that results when x is probabilistically encoded as y and then deterministically reconstructed as x ′ (y). These definitions of Q(x|y) and D VQ allow D to be written as where the second term is constant, and the third term may be ignored if y can occupy much fewer states than x (given y) can. The conditions under which the third term can be ignored are satisfed in a neural network, because x is an activity pattern, and y as the vector of locations of a finite number of firing events.
The first term of D is proportional to D VQ , whose properties may be investigated using the techniques in [3]. Assume that there are n firing events, so that y = (y 1 , y 2 , · · · , y n ), then the marginal probabilities of the symmetric part S [Pr(y|x)] of Pr(y|x) under interchange of its (y 1 , y 2 , · · · , y n ) arguments are given by where Pr(y 1 |x) may be interpreted as the probability that the next firing event occurs on neuron y (given x), Also define 2 useful integrals, D 1 and D 2 , as where x ′ (y) is any vector function of y (i.e. not necessarily related to x ′ (y)), to yield the following upper bound on D VQ where D 1 is non-negative but D 2 can have either sign, and the inequality reduces to an equality in the case n = 1. Thus far nothing specific has been assumed about Pr(y|x), other than the fact that it contains no temporal information, so the upper bound on D VQ applies whatever the form of Pr(y|x).
If the firing events occur independently of each other (given x), then Pr(y 1 , y 2 |x) = Pr(y 1 |x) Pr(y 2 |x), which allows D 2 to be redefined as where D 2 is non-negative.
In summary, the assumptions which have been made in order to obtain the upper bound on D VQ in equation 8 with the definition of D 1 as given in equation 7 and D 2 as given in equation 9 are: no temporal information is included in the network state vector (x, y), y can occupy much fewer states than x (given y) can, and firing events occur independently of each other (given x). In reality, there is always temporal information available, and the firing events are correlated with each other, so a more realistic objective function could be constructed. However, it is worthwhile to consider the consequences of equation 8, because it turns out that it leads to many non-trivial results.
The upper bound on D VQ may be minimised with respect to all free parameters in order to obtain a least upper bound. In the case of independent firing events, the free parameters are the x ′ (y) and the Pr(y|x). These two types of parameters cannot be independently optimised, because they correspond to the generative and recognition models implicit in the neural network, respectively.
A gradient descent algorithm for optimising the parameter values may readily be obtained by differentiating D 1 and D 2 with respect to x ′ (y) and Pr(y|x). Given the freedom to explore the entire space of functions Pr(y|x), the optimum neural firing behaviour (given x) can in principle be determined, and in certain simple cases this can be determined by inspection. If this option is not available, such as would be the case if biological contraints restricted the allowed functional form of Pr(y|x), then a limited search of the entire space of functions Pr(y|x) can be made by invoking parametric model of the neural firing behaviour (given x).

Examples
In this section several examples are presented which illustrate the use of D 1 + D 2 in the optimisation of discretely firing neural networks. In section 3.1 a topographic mapping network is derived from D 1 alone, in section 3.2 Pr(y|x) that minimises D 1 + D 2 is shown to be piecewise linear, and a solved example is presented. Finally, in section 3.3 a more detailed worked example is presented, which demonstrates how a factorial encoder emerges when D 1 +D 2 is minimised.

Topographic Mapping Neural Network
When an appropriate from of V VQ is considered, it can be seen that it leads to a network that is closely related to Kohonen's topographic mapping network [1].
The derivation of a topographic mapping network that was given in [2] will now be recast in the framework of section 2.2. Thus, consider the objective function for a 3-layer network (x, y, z), in which (compare equation 1) where the cost of encoding y has been ignored, so that effectively only a 2-layer network (x, z) is visible, and D VQ is given by This expression for D VQ explicitly involves (x, z), but it may be manipulated into a form that explicitly involves (x, y). In order to make simplify this calculation, D VQ will be replaced by the equivalent objective function Now introduce dummy integrations over y to obtain Pr(z|y) (13) and rearrange to obtain where Pr(y ′ |y) =

Mz z=1
Pr(y ′ |z) Pr(z|y) Pr(y ′ |y) Pr(y|x) which may be replaced by the equivalent objective function By manipulating D VQ from the form it has in equation 11 to the form it has in equation 16, it becomes clear that optimisation of the (x, z) network involves optimisation of the (x, y ′ ) subnetwork, for which an objective function can be written that uses a Pr(y ′ |x) as defined in equation 15. When optimising the (x, y ′ ) subnetwork, Pr(y ′ |y) takes account of the effect that z has on y.
If n = 1, so that only 1 firing event is observed, then D VQ = D 1 , and the optimum Pr(y|x) must ensure that y depends deterministically on x, so that Pr(y|x) = δ y,y(x) where y(x) is an encoding function that converts x into the index of the neuron that fires in response to x. This allows D VQ to be simplified to where Pr(y ′ |y(x)) is Pr(y ′ |y) with y replaced by y(x). Note that if Pr(y ′ |y) = δ y,y ′ then D VQ reduces to the objective function 2´dx Pr(x) x − x ′ (y(x)) 2 for a standard vector quantiser (VQ).
The optimum y(x) is given by y(x) = arg min y My y ′ =1 Pr(y ′ |y) x − x ′ (y ′ ) 2 (which is not quite the same as the y(x) = arg min y x − x ′ (y) 2 used by Kohonen in his topographic mapping neural network [1]), and a gradient descent algorithm for updating x ′ (y ′ ) is x ′ (y ′ ) −→ x ′ (y ′ ) + ε Pr(y ′ |y(x)) (which is identical to Kohonen's prescription [1]). The Pr(y ′ |y) may thus be interpreted as the neighbourhood function, and the x ′ (y ′ ) may be interpreted as the weight vectors, of a topographic mapping. Because all states y that can give rise to the same state z (as specified by Pr(z|y)) become neighbours (as specified by Pr(y ′ |y) in equation 15), Pr(y ′ |y) includes a much larger class of neighbourhood functions than has hitherto been used in topographic mapping neural networks. Because of the principled way in which the topographic mapping objective function has been derived here, it is the preferred way to optimise topographic mapping networks. It also allows the objective function to be generalised to the case n > 1, where more than one firing event is observed.

Piecewise Linear Probability of Firing
The optimal Pr(y|x) has some interesting properties that can be obtained by inspecting its stationarity condition. For instance, the Pr(y|x) that minimise D 1 + D 2 will be shown to be piecewise linear functions of x.
Thus, functionally differentiate D 1 + D 2 with respect to log Pr(y|x), where logarithmic differentation implicitly imposes the constraint Pr(y|x) ≥ 0, and use a Lagrange multiplier term L ≡´dx ′ λ(x ′ ) M y ′ =1 Pr(y ′ |x ′ ) to impose the normalisation constraint M y=1 Pr(y|x) = 1 for each x, to obtain The stationarity condition implies that = 0, which may be used to determine the Lagrange multiplier function λ(x). When λ(x) is substituted back into the stationarity condition itself, it yields There are several classes of solution to this stationarity condition, corresponding to one (or more) of the three factors in equation 19 being zero.
1. Pr(x) = 0 (the first factor is zero). If the input PDF is zero at x, then nothing can be deduced about Pr(y|x), because there is no training data to explore the network's behaviour at this point.
2. Pr(y|x) = 0 (the second factor is zero). This factor arises from the differentiation with respect to log Pr(y|x), and it ensures that Pr(y|x) < 0 cannot be attained. The singularity in log Pr(y|x) when Pr(y|x) = 0 is what causes this solution to emerge.

3.
M y ′ =1 (Pr(y ′ |x) − δ y,y ′ ) x ′ (y ′ ) · (· · · ) = 0 (the third factor is zero). The solution to this equation is a Pr(y|x) that has a piecewise linear dependence on x. This result can be seen to be intuitively reasonable because is a linear combination of terms of the form x i Pr(y|x) j (for i = 0, 1, 2 and j = 0, 1, 2), which is a quadratic form in x (ignoring the x-dependence of Pr(y|x)). However, the terms that appear in this linear combination are such that a Pr(y|x) that is a piecewise linear function of x guarantees that f (x) is a piecewise linear combination of terms of the form x i (for i = 0, 1, 2), which is a quadratic form in x (the normalisation constraint M y=1 Pr(y|x) = 1 is used to remove a contribution to that is potentially quartic in x). Thus a piecewise linear dependence of Pr(y|x) on x does not lead to any dependencies on x that are not already explicitly present in D 1 + D 2 . The stationarity condition on Pr(y|x) (see equation 19) then imposes conditions on the allowed piecewise linearities that Pr(y|x) can have.
For the purpose of doing analytic calculations, it is much easier to obtain analytic results with the ideal piecewise linear Pr(y|x) than with some other functional form. If the optimisation of Pr(y|x) is constrained, by introducing a parametric form which has some biological plausibility, for instance, then analytic optimum solutions are not in general possible to calculate, and it becomes necessary to resort to numerical simulations. Piecewise linear Pr(y|x) should therefore be regarded as a convenient theoretical laboratory for investigating the properties of idealised neural networks.

Solved Example
A simple example illustrates how the piecewise linearity property of Pr(y|x) may be used to find optimal solutions. Thus consider a 1-dimensional input coordinate x ∈ [−∞, +∞], with Pr(x) = P 0 . Assume that the number of neurons M tends to infinity in such a way that there is 1 neuron per unit length of x, so that Pr(y|x) = p(y − x), where the piecewise linear property gives p(x) as and by symmetry x ′ (y) = y. This Pr(y|x) and x ′ (y) allow D 1 to be derived as and D 2 to be derived as Because there is one neuron per unit length, the contribution per unit length to D 1 + D 2 is the sum of the above two results If D 1 +D 2 is differentiated with respect to s, then stationarity condition d(D1+D2) ds = 0 yields the optimum value of s as and the stationary value of D 1 + D 2 as When n = 1 the stationary solution reduces to s = 0 and D 1 + D 2 (per unit length) = P0 6 , which is a standard vector quantiser with nonoverlapping neural response regions which partition the input space into unit width quantisation cells, so that for all x there is exactly one neuron that responds. Although the neurons have been manually arranged in topographic order by imposing Pr(y|x) = p(y − x), any permutation of the neuron indices in this stationary solution will also be stationary solution. This derivation could be generalised to the type of 3-layer network that was considered in section 3.1 , in which case a neighbourhod function Pr(y ′ |y) would emerge automatically.
As n −→ ∞ the stationary solution behaves as s −→ 1 2 and D 1 + D 2 (per unit length) −→ P0 3n , with overlapping linear neural response regions which cover the input space, so that for all x there are exactly two neurons that respond with equal and opposite linear dependence on x. As n −→ ∞ the ratio of the number of firing events that occur on these two neurons is sufficient to determine x to O 1 n . When n = ∞ this stationary solution is s = 1 2 and D 1 + D 2 (per unit length) = 0. However, when n = ∞ there are infinitely many other ways in which the neurons could be used to yield D 1 + D 2 (per unit length) = 0, because only the D 2 term contributes, and it is 0 when x = M y=1 Pr(y|x) x ′ (y). This is possible for any set of basis elements x ′ (y) that span the input space, provided that the expansion coefficients Pr(y|x) satisfy Pr(y|x) ≥ 0. In this 1dimensional example only two basis elements are required (i.e. M = 2), which are x ′ (1) = −∞ and x ′ (2) = +∞. More generally, for this type of stationary solution, M = dim x + 1 is required to span the input space in such a way that Pr(y|x) ≥ 0, and if M < dim x + 1 then the stationary solution will span the input subspace (of dimension M − 1) that has the largest variance.
The n = 1 and n −→ ∞ limiting cases are very different. When n = 1 the optimum network splits up the input space into non-overlapping quantisation cells, and as n −→ ∞ the optimum network does a linear decomposition of the input space using non-negative expansion coefficients. This behaviour occurs because for n > 1 the neurons can cooperate when encoding the input x, so that by allowing more than one neuron to fire in response to x, the encoded version of x is distributed over more than one neuron. In the above 1-dimensional example, the code is spread over one or two neurons depending on the value of x. This cooperation amongst neurons is a property of the coherent part D 2 of the upper bound on D VQ (see equation 8).

Factorial Encoder Network
For certain types of distribution of data in input space the optimal network consists of a number of subnetworks, each of which responds to only a subspace of the input space. This is called factorial encoding, where the encoded input is distributed over more than one neuron, and this distributed code typically has a much richer structure than was encountered in section 3.2.
The simplest problem that demonstrates factorial encoding will now be investigated (this example was presented in [4], but the derivation given here is more direct). Thus, assume that the data in input space uniformly populates the surface of a 2-torus S 1 × S 1 . Each of the S 1 is a plane unit circle embedded in R 1 × R 1 and centred on the origin, and S 1 × S 1 is the Cartesian product of a pair of such circles. Overall, the 2-torus lives in a 4-dimensional input space whose elements are denoted as x = (x 1 , x 2 , x 3 , x 4 ), where one of the circles lives in (x 1 , x 2 ) and the other lives in (x 3 , x 4 ). These circles may be parameterised by angular degrees of freedom θ 12 and θ 34 , respectively.
Assume that the encoding functions partition up the 2-torus symmetrically, the three types of solution may be optimised as described in the following three sections. stable), type 2 (intermediate), type 1 (least stable). Similarly, for constant n and letting M −→ ∞, the relative stability of the three types of solution is: type 2 (most stable), type 3 (intermediate), type 1 (least stable).
In both of these limiting cases the type 1 solution is least stable. If there is a fixed number of firing events n, and there is no upper limit on the number of neurons M , then the type 2 solution is most stable, because it can partition the 2-torus into lots of small quantisation cells. If there is a fixed number of neurons M (which is the usual case), and there is no upper limit on the number of firing events n, then the type 3 solution is most stable, because the limited size of M renders the type 2 solution inefficient (the quantisation cells would be too large), so the 2-torus S 1 × S 1 is split into two S 1 subspaces each of which is assigned a subset of M 2 neurons. If n is large enough, then each of these two subsets of neurons has a high probability of occurrence of a firing event, which ensures that both of the S 1 subspaces are encoded.
More generally, when there is a limited number of neurons they will tend to split into subsets, each of which encodes a separate subspace of the input. The assumed form of Pr(y|x) in equation 26 does not allow an unrestricted search of all possible Pr(y|x). If the global optimum solution (which has piecewise linear Pr(y|x), as proved in section 3.2) cuts up the input space into partially overlapping pieces, then it is well approximated by a solution such as one of those listed in equation 26. Typically, curved input spaces lead to such solutions, because a piecewise linear Pr(y|x) can readily quantise such spaces by slicing off the curved "corners" ' that occur in such spaces.

Conclusions
In this paper an objective function for optimising a layered network of discretely firing neurons has been presented, and three non-trivial examples of how it is applied have been shown: topographic mapping networks, piecewise linear dependence on the input of the probability of a neuron firing, and factorial encoder networks. Many other examples could be given, such as combining the first and third of the above results to obtain factorial topographic networks, or extending the theory to multilayer networks, or introducing temporal information.

Acknowledgements
I thank Chris Webber for many useful conversations that we had during the course of this research.