{'fraction_non_alphanumeric': 0.027684383718434794, 'fraction_numerical': 0.009728879032834967, 'mean_word_length': 4.609951845906902, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Novel Spatial Architecture Artificial Neural Network Based on Multilayer Feedforward Network with Mutual Inhibition among Hidden Units

We propose a Spatial Artificial Neural Network (SANN) with spatial architecture which consists of a multilayer feedforward neural network with hidden units adopt recurrent lateral inhibition connection, all input and hidden neurons have synapses connections with the output neurons. In addition, a supervised learning algorithm based on error back propagation is developed. The proposed network has shown a superior generalization capability in simulations with pattern recognition and non-linear function approximation problems. And, the experimental also shown that SANN has the capability of avoiding local minima problem.


Introduction
Warren McCulloch and Walter Pitts [1] gave an academic background of applying artificial neural networks (ANNs) because they have shown that ANNs have the capability of computing any arithmetic or logical function in principle. Because of the characteristic of resembling the process we go through learning something [2], good at providing fast and close approximations of the correct answer, and best at identifying patterns and trends in data, ANNs have already been successfully applied in many industries and daily life for prediction and forecasting [3][4][5][6][7][8].
Theoretical research of ANNs major focus on two points: architecture (BP, Hopfield network, RBF etc.) and learning algorithm (Back propagation, Levenberg-Marquardt, Conjugate Gradient algorithm etc.). These structures of ANNs basically are planar and symmetrical connection. But the animal's neuronal system is there dimensional and asymmetry; and the neurons of brain in previous layers have narrow perception field but have wider perception fields in deep layers. On the other hand, structure defines function in ANNs. So, there are some structure studies, for instance, based on functional expansion, Chao proposed a new pyramid network [9] and analyzed the representation and generalization theoretically and experimentally; Jaeger presented Echo State Networks [10], which combined the trainable linear combination of nonlinear response signals of internal units and the desired output signals. Research of cortex shown that the inhibition of cerebral cortex may avoid the excitatory activity caused by a stimulus ripple across the entire network and prevent the confused jumble resulted by the overlapping signals [11].
We proposes a neural network with new spatial architecture called Spatial Artificial Neural Network (SANN) to tradeoff the structure and functions and aim to achieve higher representation and generalization capability. SANN is based on a multilayer feed-forward network, spatial span connection and recurrent lateral inhibitory connection mechanism. Through SANN, all non-output neurons have connections to the output neurons. Supervised training rules of SANN are then shown for weights update and learning.
The proposed network applied to several benchmark problems indicated that this network yields significantly representation capability and generalization performance, and it also can avoid the local minima problems in the supervised learning.

Definition of Spatial Artificial Neural Network
We define the spatial artificial neural network as follows, and the topology structure is shown in Fig. 1. The first part of SANN architecture is so-called the basic network lies in the center of SANN architecture. Actually the basic network is a multilayer feedforward network which adds mutual inhibition mechanism among the hidden units within a same layer. The feedforward connections among neurons are shown in Fig. 1 using arrowhead line, and its mutual inhibition adopts the recurrent lateral inhibition mechanism, as shown in Fig. 2. The second part of SANN is the spatial span connections part which is shown with arrowhead broken lines. This basic network is used for transferring information from the environmental inputs to the network outputs. The recurrent lateral inhibitory connection of one hidden layer brings, which is widely found in biological neural systems, connection and competition to the hidden units.
The input signals were transferred to hidden units through the feedforward connections, and the outputs of hidden units corresponding to these input signals are called local output. And in the lateral inhibition theory, it is also called activation level. It will inhibit other hidden units with strength of inhibitory coefficients. The feedforward input combines with the inhibitory input (caused by other hidden units and calculated according to the product of the inhibiting neuron's activation level and the corresponding recurrent lateral inhibitory coefficient) as the total input of hidden units, and the output was transferred to the next layers as the input signal.
The accessorial spatial span connections lay over the basic network of SANN. This part make that any two units of SANN in different layers may have direction connect according to the spatial span connection and weights. The global output of any unit through this span connection transfer signals to the next two or more layer neurons.
Both the inhibitory connection of hidden units and the spatial span connection between different layers guarantee the structure of SANN is a there dimensional and asymmetry system, which resembles the biological neuronal system structure.

Review of Lateral Inhibition
Lateral inhibition (LI) is one type of inhibitions of cerebral cortex. It is one of the basic principles of information treatment within neural system and describes the capacity of an excited neuron to reduce the activity of its neighbor neurons. In the design of our SANN network, the architecture in hidden layers uses the recurrent lateral inhibition to achieve competition.
The topology of lateral inhibitory connection without self-inhibition adapted in SANN is show in Fig. 2, and the mathematic model is given by [12] where h is the number of hidden units, y o,j is the output of j -th hidden neuron, y i,j is the environment stimulates of neuron j received, v rj = v jr is the lateral inhibitory coefficient between neuron r and j, θ rj is the inhibiting threshold value of neuron j due to neuron r.

Mathematic Model of SANN
Suppose the SANN has L+1 layers including the input and output layers, the l -th (l =1, 2, ..., L-1) hidden layer has n l units. The input layer is denoted as The output of the k -th unit in the output layer is given by: where the f l is the activation function, o jl is the output of the j -th hidden neuron of the l -th layer, it is given by: where v rj is the inhibitory coefficient from neuron r to neuron j, o r l is the local output of hidden unit r in l -th layer, namely the activation level of neuron r, it is given by: From Fig. 1 and Eq. (4) we can find that it is output of the current hidden neuron due to the feedforward input signals, namely the output comes from the input neurons. v rj is the recurrent lateral inhibitory weight between neuron r and j of hidden layer l.

Learning Algorithm
Here, we reference the standard back propagation algorithm to derivate the learning algorithm fit for SANN. It is described as follows.
Assume the three-layer SANN with n input neurons, h hidden neurons and one output neurons respectively. The output of the output unit k due to the p-th (p ∈ [1, P ]) input sample is given by o pk , whereas the desired output is t pk , and the output of the j -th hidden unit for the p-th input pattern is given by o p j . Let ω jk be the weight between the k -th output unit and the j -th hidden unit, and ω ij be the weight between the the j -th hidden unit and i-th input unit. The input for the i-th input unit due to the p-th input pattern is denoted by x pi .
According to the above definitions, the output of the j -th neuron in the hidden layer is given by: where,ō pr = f ( n i=1 ω ij x pi ) is the output of the r -th hidden neuron due to the feedforward input signals, namely the output comes from the input neurons; v rj is the recurrent lateral inhibitory weight between neuron r and j of hidden layer; f is the sigmoid activation function defined as: Similarly, the output of the k -th unit in the output layer is given by: We define the sum of squared error of output unit with P samples as the system learning target function, to be: The learning algorithm based error back propagation is to change the weights iteratively such that the function E Eq. (7) is minimized. The weight updates are proportional to ∂E. According to the chain rule, we can obtain the partial derivative of ∂E with respect to w jk and w ik , and the weight change for the (t +1)-th iteration can be expressed as follows, (8) where the term μ is the learning rate of the gradient method.
It is worthwhile to notice that the above gradient consists two parts, the first term is the basic back propagation in the basic network and the second term is the span back propagation in the spatial span connection network.
The update of SANN weights according to ω sk (t+1) = ω sk (t)+αΔω sk . Where the term α is the momentum term used to balance the stability and oscillate which may caused by the value selection of learning rate μ.

Numerical Experimental
We use two benchmark problems to compare the performance of SANN using the above supervised learning rule to train and the multilayer feed-forward network is trained by implementation of the back propagation algorithm in Matlab.
In our experiments, 10-fold cross validation is employed on each problem to compare the performance. The result is the average result of the ten folds.

Classification Capability (XOR Problem)
In order to consider the classification capability of SANN, we chose a simplest problem which is not linearly separable -"exclusive-or" problem which is also discussed in [13][14][15]. According to the complexity of XOR problem, in this section we consider to train 2-1-1 and 2-2-1 two architectures of SANN on the two dimensional XOR classification problem described in the first two rows of Table  1. Note that the inputs can still be 0 and 1 but the desired values must be changed keeping in mind the signal range. The corresponding outputs of two experiments are as shown in the last two rows of Table 1.  Fig. 3 shows the training errors of the network output and desired output in the supervised algorithm. The test input samples and output as shown in Table 2. The average result of ten folds for SANN to solve the XOR problem with architecture 2-1-1 of τ = 0 are as follows: running time is 0.0812 s, training mse is 1.77919e-25, test mse is 1.89882e-25. From Table 1-2, and Fig. 3 we can find that a SANN network with only one hidden unit can solve the xor problem at any training accuracy, meanwhile, the similar test accuracy was obtained. Besides, the convergence is very fast that it only needs several iterations.

(2) 2-2-1 BPN and SANN
The smallest architecture of multilayer feedforward network is with two hidden units. Here, we compare the generalization capability of SANN and BPN with the architecture of 2-2-1.
Set the value of global parameters are as follows: max iteration number T = 2000, learning rate μ = 0.8, moment term α = 0.7, and the error tolerances τ = 0. The training mse is shown in Fig. 4, and the average result of ten folds for SANN and BPN respectively is given by in Table 3. From this experiment we can find that when they both have two hidden neurons to solve the xor problem, the SANN network has the similar training and test accuracy with BP network, and the convergence of SANN network is faster than BP network, but it need more training time because there are more weights should be calculated than BP network in the learning.

400
. Where x i (i = 1, 2, ..., 8) is drawn at random from [0, 10]. Similar mapping functions have been used in [16][17][18]. In this paper we use 250 samples to train a SANN network of size 8-4-1 (same with [17]). The parmaters are set as: the learning error tolerance τ = 1e − 5, max iteration number T = 1000, learning rate μ = 1e − 4, moment term α = 0.7. The training data set are sampled uniformly in the domain. In order to estimate the generalization capability of SANN network, an independent set of 500 test data is generated at random. The weights are initialized as the  Table 4 lists the comparison results of the SANN network and BP network. The training error results are shown in Fig. 5.
Besides, we also use a SANN network with two hidden units to approximate the nonlinear function, the training error is shown in Fig. 6.
In this experiment, we can find that for a given training accuracy and finite iteration number, SANN network need little hidden units than multilayer feed-forward network. Whether has the same number of hidden units or less than, SANN network performs better than the multilayer feedforward network in terms of both the approximation capability and generalization capability. Besides, SANN can alleviate or avoid the local minima problem.

Conclusion and Discussion
In this paper, a novel topology architecture of artificial neural network is proposed based on multilayer feedforward network, lateral inhibition mechanism and spatial span connection mode, and named SANN. The activation function of input neurons is tunable and according to the priori knowledge to chose a proper function as the input neurons' activation function. In this paper, when they connect to the output neurons, choose a nonlinear function, otherwise, choose a linear function. The proposed network shown higher representation and generalization capability in the numerical experiments, and can alleviate or avoid the local minima problem than multilayer feedforward network. But because of the more synapse connections than feedforward network, it needs more time to learn the samples. Design a suitable learning algorithm for SANN is the next useful work.