{'fraction_non_alphanumeric': 0.03348651753733865, 'fraction_numerical': 0.007636683111441509, 'mean_word_length': 4.673888036809816, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Flexible Transmitter Network

Current neural networks are mostly built upon the MP model, which usually formulates the neuron as executing an activation function on the real-valued weighted aggregation of signals received from other neurons. In this paper, we propose the Flexible Transmitter (FT) model, a novel bio-plausible neuron with flexible plasticity. The FT model employs a pair of parameters to model the transmitter between neurons and sets up a neurotransmitter regulated memory unit to record the long-term learning information of the concerned neuron, thus leading to the formulation of the FT model as a two-variable two-valued function, which takes the commonly-used MP neuron model as its special case. The FT model can handle more complicated data, even time series signals. To exhibit the power and potential of our FT model, we present the Flexible Transmitter Network (FTNet), which is built in the most common fully-connected feed-forward architecture by incorporating the FT neuron as the basic building block. FTNet allows gradient calculation and can be implemented by an extension of the backpropagation algorithm in the complex domain. Experiments on a board range of tasks show the superiority of the proposed FTNet. This study provides an alternative basic building block in neural networks and exhibits the feasibility of developing artificial neural networks with neuronal plasticity.

neural network algorithms and network architectures have been developed, the modeling of neurons is relatively less considered. The most famous and commonly used formulation of neuron is the MP model [12], as illustrated in The MP model is very successful though the formulated cell behaviour is quite simple. Real nervous cells are much more complicated, and thus, exploring other bio-plausible formulation with neuronal plasticity is a fundamental problem. There have been many efforts on modeling the spiking behaviour of cells, leading to spiking neuron model and pulsed neural networks [9,18] that takes spiking neurons as basic computational units. In this work, we consider another interesting aspect and propose a novel type of neuron model.
Neuroscience studies [2,7] disclose that the synapses ensure the one-way communication mechanism between two neurons, that is, the information flow is from pre-synaptic cells to postsynaptic cells. Synapses usually form between the axons of pre-synaptic cells and the dendrites of postsynaptic cells. In common synaptic structures, there is a gap (called synaptic cleft in neuroscience) of about 20 µm between the dendrites and the axons, as shown in Figure 2. This means the axonal transmission strength of pre-synaptic cells and the dendritic concentration of postsynaptic cells are different, albeit closely related. Therefore, it is naturally necessary to distinguish between the pre-and post-synaptic portions in a neuron model. Contrary to the MP model that simply treats the entire synaptic structure as a real number w and spiking neuron that characterize as an ODE equation of leaky integration, in this paper, we use a pair of related parameters (w, v) to represent the axonal transmission strength and dendritic concentration, respectively, that is, flexible transmitters. Besides, lots of experimental studies [8,6] point out that neurons have memories of past learning behaviours, and the biological voltage can be persistent strengthening or weakening based on recent patterns of activity, that is, the Long-Term Potentiation or Depression. As memories are thought to be encoded by modification of voltage strength, we in this work specifically set up a memory variable, the neurotransmitter regulated memory unit to record the influence of long-term learning behaviours.
According to the neuron illustration in Figure 2, the axon portion only transmits the output signal of neuron B, while the dendritic concentration is not only stimulated by the received signal, but also related to the memory of neuron A itself. Inspired by this, we resort the two types of signals associated with a synapse -the stimulus transmitted between neurons and the memory unit of the concerned neuron to a paired formation, leading to a formulation of this neural learning behaviour by a two-variable two-valued function.
In Section 2, we will introduce the Flexible Transmitter (FT) model in detail.
Regard the fundamental FT neuron as the basic building block, various network architectures can be tried, whereas the simplest may be the full-connected feed-forward architecture that has been popularly applied with MP model. In order to demonstrate the power and potential of our FT model, in Section 3, we present the Flexible Transmitter Network (FTNet), which is a full-connected network constructed by replacing the real-valued MP model with FT neuron. Correspondingly, a practicable and effective backpropagation algorithm for training FTNet is developed. Experiments on broad range spatio-temporal data sets have been conducted in Section 4. The results show that FTNet is able to get excellent performance by using the same default setting. Finally, we conclude our work in Section 5.

Flexible Transmitter Model
The interesting discovery of neuroscience in Figure 2 suggests that, the response of neuron A to the stimulus signals received from neuron B depends on not only neuron B's axonal transmission strength but also neuron A's dendritic concentration related to the memory unit of neuron A.
Inspired by this recognition, we propose the FT model, as illustrated in Figure 3. In contrast to the MP model that the interaction between two neurons is formulated by a single connection weight, in FT model the interaction comprises two parts: wx t where x t is the current stimulus sent to the concerned neurons via the corresponding axonal transmission strength w, and vm t−1 where m t−1 is the memory intensity of the concerned neuron at (t − 1)-th timestamp related to the dendritic concentration v. In other words, the FT model employs a pair of transmitter parameters (w, v) rather than a real-valued weight w in the MP model to indicate the synaptic plasticity. On the other hand, the output of FT neuron at the t-th timestamp also consists of two parts: s t and m t , where s t is the bio-electric/ chemical stimulus generated by the concerned neuron and m t indicates the current memory intensity of the concerned neuron. After each timestamp, the stimulus signal s t will be transmitted to the next neurons, while the memory intensity of the concerned neuron will be renewed by m t .
This modeling approach can lead to many benefits. Firstly, paired parameters separate the roles of axonal transmission strength and dendritic concentration. The axons transmit external stimuli, while the dendrites connect to the memory units of the concerned neuron. FT model provides greater flexibility for neuronal plasticity, making the MP model as its special case. Secondly, the FT model employs a neuron-exclusive variable m t to indicate the neurotransmitter regulated memory intensity. During the learning process, the memory unit achieves self-renewal, thus deriving a local recurrent system. From a neuroscience perspective, we can think of this local recurrent system as simulating the learning-and-memory process of neurons.
Therefore, the FT model can handle more complicated data, even time series signals.
In summary, the proposed FT model employs a pair of parameters (w, v) to indicate the synaptic plasticity and puts up a neuron-exclusive variable m t to represent the neurotransmitter regulated memory unit.
This modeling makes the FT neuron not only be more biologically realistic, but also be more potential and powerful for dealing with complicated data. Furthermore, the FT model spontaneously leads to a formalization with a two-variable two-valued function f and a pair of parameters (w, v), that is, We call this model Flexible Transmitter.

An Implementation of FT Model
According to Equation 1, the FT model is inherently dominated by a two-variable two-valued function f and a pair of parameters (w, v). Both the input and output of the FT model comprise two parts, and their relationship would be quite complicated since the output variables s t and m t share common parameters (w, v) and input (x t , m t−1 ). Most existing neuron models are dependent on one-valued functions, which are hard to be directly applied to this concern. An interesting solution is to resort to a complex-valued formulation that represents the input and output of the concerned neurons, respectively, leading to the neuron model as follows: In complex analysis, the real and imaginary parts of the output of a complex-valued function are geminous twins; both s t and m t share the same one complex-valued function f and common parameters (w, v). In other words, m t can keep in touch with s t by an underlying law during the learning process. So given a specific function f , if mastered the value or formulation of s t , we could easily derive m t . Further, once the stimulus s t is supervised by some teacher signals, even if leaving memory intensities m t unsupervised, m t can still be corrected according to the twins' law.
Finally, we emphasize that using complex-valued function is just one kind of approach for implementing the proposed FT model. It may not be the most appropriate one, and it is very likely that there are better approaches to be explored in the future.

A Simple Architecture of FT Net
The FT neuron is a fundamental unit of neural networks. To evaluate its potential, we consider to use the simplest full-connected feed-forward neural network through replacing original MP neuron by the FT neuron as its basic building block, thus we get the FTNet. Based on Equation 2, we can provide a general vectorized representation for a layer of FT neurons: It is worth noting that if considering m-dimensional external input signals X t and n-dimensional output stimuli S t , then the transmitter concentration matrix W ∈ R n×m and V is a n-dimensional square

Holomorphic Conversion Function
In order to perform backpropagation in a FTNet, it's necessary for the conversion function to be differentiable, i.e., holomorphic. The detailed information about holomorphism can be obtained in Appendix A. By the holomorphism, we restricted the set of possible conversion functions. There are various holomorphic functions can be tried. Ideally, we can get inspiration from bio-science to design this conversion function; here we have not incorporated bio-knowledge and simple use the simplest linear holomorphic function, where α and β are constants in R.

Activations
Next, we are going to introduce some activation functions that can generally be used in FTNet. An intuitive idea is to decompose the activation function σ into two real-valued nonlinear functions that are activated with respect to the real and imaginary parts, respectively, that is, σ = σ real + σ imag i, where σ real and σ imag are real-valued activation functions, such as the sigmoid, tanh, or softmax functions.
Apart from this, FTNet also allow non-trivial activations in the complex domain. Indeed, efforts have been made in complex-valued activations, such as the modReLU [1] and zReLU [17].
where z ∈ C, θ z is the phase of z, and b ∈ R is a learnable parameter.
Here, we propose an alternative complex-valued ReLU activation, the Polar ReLU (PReLU). PReLU is a point-wise nonlinear function, which limits the radius and angle of a complex number in the polar coordinate system, defined as: ρ is a learnable or predefined parameter, indicating the minimum radius of the complex value z, θ z is the phase of z, and θ * restricts the allowed excitation angle. Figure 4 illustrates the working mechanisms of these aforementioned complex-valued activations from the perspective of geometric manifolds. The modReLU function creates a "dead circle" with a learnable radius b, while zReLU emphasizes the allowed excitation phase of z. Our proposed PReLU is able to juggle both angle and radius. As |z| is always greater than ρ, PReLU could create a "dead circle" with radius ρ and centre (0, 0). Additionally, PReLU also restricts the allowed excitation angel by a pre-given θ * . Thus, PReLU has greater flexibility and potential, and becomes an awesome indicator to evaluate the importance of radius and phase in complex-valued activations. In addition, for further comparison experiments on activation, see Subsection 4.1.
By employing the holomorphic conversion and complex-valued activation functions, a complete FTNet is established. Throughout this paper, we will use the notations FT0 to denote the FTNet without any hidden layer, i.e., size(m, n) or size(m, 0, n) and FT1 to indicate the FTNet with FTNet with only one hidden layer corresponding to size(m, h, n), where h is the number of hidden neurons.

Complex Backpropagation
In this section, we present the Complex Backpropagation (CBP) algorithm for training the FTNet. CBP is an extension of the common backpropagation algorithm in the complex domain. Two concrete implementation demos for CBP with one and two layers FTNets are token in Appendix B.
Before that, we re-emphasized the notations in FTNet. We might consider a L-layer FTNet, where the l-th layer has n l neurons. By employing the linear conversion function and activation functions mentioned in Section 3, the feed-forward procedure of FTNet becomes: where S l t and M l t denote the stimuli signals and memory intensities in l-th hidden layer at time t, respectively, σ indicates the activation function, which can be divided into the real part σ real and imaginary part σ imag , and S 0 t = X t . For convenience, we use shorthand notations Real l t and Image l t for denoting the input terms of FT neurons, that is, Considering the loss function for FTNet in time interval [0, T ], where Y t = S L t is the stimulus signals of the final hidden layer and Y true t is the corresponding supervised signals. Taking into account the temporal dependency, the backpropagation gradients of transmitter concentration matrices through time can be calculated by: The aforementioned formula consists of three terms. (1) The first term δS l t is the backpropagation error correction in the l − th layer at time t. So we can calculate it by is the core of our CBP algorithm, including two backpropagation pipelines with respect to W l t and V l t , respectively.
In the feed-forward process, the stimulus signals and memory intensities are outputted by holomorphic FT neurons with coupling processing capability; while in the CBP process, the memory intensities are indirectly regulated by the supervised stimulus signals. So we still need to supply the feed-forward errors caused by the memory intensities. The calculation derivatives are similar to those of Therefore, we directly provides the results: . Until the previous step, the CBP algorithm has corrected the network parameters based on the supervisory stimuli (at the last point in time), leaving the imaginary part of the final complex signal uncorrected. So we still need to update the imaginary parts (memory intensities) before using the FT network for prediction: And then, the feed-forward errors caused by the unsupervised memory intensities can be believed to vanish, thus the elements of ∂ M t−1 ∂ W and ∂ M t−1 ∂ V should be cleared to 0.

Discussion
The proposed FT model puts up the neuron-exclusive variable to record the long-dependent memory information about spatio-temporal signals. In addition, as the memory unit renew by itself during the learning process, there induces a brand-new local recurrent system. Different from the basic recurrent block of RNNs [19] formalized by a real-valued function s t = g(x t , s t−1 ; w, v), the FT model is dominated by (s t , m t ) = f (x t , m t−1 ; w, v). Obviously, the RNNs's unit is a special case of the FT model; these two models are equivalent to each other when pre-setting s t = m t . Therefore, with the same number of parameters w and v, the FT model is able to generate more features, both s t and m t , leading to a broader neuronal representation.
In order to further demonstrate its superiority thoroughly, we design a simple experiment as follows. The data set is a mixture of a cosine function and a sine function with a period of 3 over 300 timestamps. We take only one MP model, one RNNs' unit, and one FT neuron respectively to fit the aforementioned mixing curve, and then appraise their performance. Here, we use the complex-valued function to implement the FT model, such as Equation 2. The comparative results shown in Figure 5(a) state that our FT neuron outperforms the other two models, achieving the minimum MSE. It is proved that our FT model can handle more complicated data rather than the conventional MP model and RNNs.
As mentioned above, the real and imaginary parts of a complex-valued function' output obey the twins' law, that is, s t and m t in Equation 2 are geminous twins. To illustrate this view, we also investigated the prediction signals and memory records of the FT neuron, which are shown in Figure 5(b). Compared to the prediction signals, the memory records evolve with smaller amplitudes but always maintain the correct directions. In other words, the imaginary memory variable becomes an "adjoint" one of the real prediction.
When FT neuron makes feed-forward forecasting, the "adjoint" variable would play an auxiliary role and contribute to improving precise performance. The detailed information of the "adjoint" mechanism is provided in Appendix A.

Experiments
In this section, we compare FTNet with several popular neural networks on three simulated and practical data sets. The goal is to demonstrate its superior performance.

Simulated Signals
We first explore the performance of our FTNet with different configurations on simulated data. The simulation data is generated by aggregating five cosine functions (each function with a period of 3-7) over 900 timestamps. For practice, every component cosine function was fused with a noise signal uniformly sampled with 15% -30% amplitude, which are illustrated in Figure 6(a). The supervised signals are the mixture of these cosine functions without noise. And we trained FTNet with the first 800 points (blue) and forecast the future 100 points (red).
We ran the FT0 net with five activation functions (sigmoid, tanh, modReLU , zReLU , and P ReLU ) until convergence on the test data. Among that, the radius and phase of these complex-valued activations are preset to 0.3 and [0, π/2], respectively. The experimental results that evolve as training iteration increases are plotted in Figure 6(b). All the FTNets except the sigmoid function perform laudably well; however, tanh performs better, achieving the best MSE. Besides, modReLU has unstable performance, while zReLU performs better than P ReLU . Considering the influence of the cascade structure, we copy this experiment on FT1 with 10 hidden neurons, and display experimental results in Figure 6(c). The tanh function also outperforms other activations, the second best is zReLU , and P ReLU is close behind. The remaining two activations perform very poorly; FTNet with the sigmoid activation seems to be strenuous to convergence, while zReLU is even invalid. Besides, we also investigate the effect of the number of hidden neurons. The results are shown in Figure 6(d). There is a split effect of neuron quantity on the performance of our FTNet when using different activation functions; most models perform better in the case of 10 hidden neurons, whereas modReLU does without the hidden layer and the change in neuron quantity has little effect on its performance. In addition, both tanh and zReLU are able to convincingly overtake the baseline 0.02. P ReLU achieves a lower MSE, but there is still a certain gap compared with the former two. The networks with sigmoid and modReLU activations have the poor performance.
In summary, we can conclude that resorting tanh as the activation function is able to attain the best performance, whether using FT0 or FT1 architecture. By contrast, the sigmoid and modReLU functions perform unfavorably. The performance of zReLU is a little better than that of P ReLU . We conjecture the reason is that the radius may be more susceptible and important than the phase for complex-valued activations. So for the FTNet configurations in the next real-world tasks, we employ tanh for activation function and 0.01 for learning rate. Other hyper-parameters cannot be fixed across tasks, otherwise, the performance will be embarrassingly unsatisfactory. Therefore, we examine a variety of configurations on the testing data set and pick out the one with the best performance.

Univariate Time Series Forecasting -Yancheng Automobile Registration
Next, we conduct experiments on the data sets of Yancheng Automobile Registration Forecasting competition 2 , which is a real-world univariate time series forecasting task. This competition requires players to use the daily automobile registration records of a certain period of time (nearly 1000 dates) in the past to predict the number of automobile registration per day for a period of time in the future. Although the actual competition allows the contestant to freely develop other data sets or information as an aid, here we only consider the data of the total number of automobile registration for 5 car brands, not including any specific date information. For simplicity, we fix the test timestamps to be the terminal subset of observations: the last 100 / 1000. This forecasting task is hard since not only objective automobile registration series is the mixture of 5 car brands, but also there exists lots of missing data and sudden changes caused by holiday or other guiding factors which we cannot obtain in advance.
2 https://tianchi.aliyun.com/competition/entrance/231641/information We compare our proposed FTNet with several state-of-the-art statistical models and neural networks [16,4], and evaluate the performance by Mean Square Error (MSE). The results are summarized in Table 1, showing that FT1 achieves highly competitive performance.

Multivariate Time Series Forecasting -Traffic Prediction on HDUK
We also validate FTNet on the Highway Data of United Kingdom (HDUK) 3   denotes a size(*,0,1) cascade structure and iterates 100 times; denotes size(*,100,1) cascade structure and iterates 100 times; ♦ indicates a network configuration with 100 recurrent neurons and 32-dimensional convolution layer) and iterates 100 epochs. Table 2 lists the comparative results of FTNet and other neural networks on a part of HDUK data sets, including the Total Traffic Flow of first 3 crossroads: A1, A1033, and A11. FTNet consistently achieves the best performance in the same settings as other competing neural networks.

Image Recognition on Pixel-by-pixel MNIST
In practice, we also conduct the experiments on the image recognition task, to evaluate the ability of FTNet for processing with spatial information. Pixel-by-pixel MNIST is a popular challenging image recognition data set, which is standard benchmark to test the performance of a learning algorithm. Following a similar setup to [1,14], the handwritten digit image is converted into a pixel-by-pixel spiking train with T = 784.
Standard split of 60,000 training samples and 10,000 testing samples was used with no data augmentation.
In general, we also add a convolutional filter to the external input signals at each time stamp. The size of the convolutional kernel is preset as 2 × 2.
In this experiment, we also compare our FTNet with another bio-inspired neural network, the spiking neural network (SNN) [15]. For classification, we use the spiking counting strategy. In other words, during training, we specify a target of 20 spikes for the true neuron and 5 spikes for each false neuron; while testing, the output class is the one which generates the highest spike count. In order to accurately compare the performance of various deep learning models, we force all networks to adopt the same setting (including 150 hidden neurons and the number of loops), except for SNN which is allowed more neurons to ensure convergence. The experimental results are shown in Table 3, confirming the superiority of our FTNet to the existing state-of-the-art neural networks.

Conclusion and Prospect
In this paper, we proposed the FT model, which is a brand-new modeling for bio-plausible nervous system.
In contrast to the traditional MP model that simply regards the entire nervous synapse as a real-valued number, FT model tries to achieve a more flexible and bio-plausible learning system, that is, considers taking neurotransmitter regulated memory units into synaptic plasticity, thus leading to a formulation of a two-variable two-valued function with a pair of transmitter parameters. To demonstrate the power and potential of our proposed FT model, we present the FTNet using the most common full-connected feed-forward architecture. For simplicity, we employ the holomorphic complex-valued reaction as an implementation paradigm, and then, present a practicable and effective CBP algorithm for training a FTNet.
The experiments conducted on wide-range tasks confirm the effectiveness and superiority of our model.

Supplementary Materials of Flexible Transmitter Network (Appendix)
In this Appendix, we provide some detailed information about the main text, constructed according to the corresponding sections therein.

A.1. Holomorphism
A complex-valued function f is called holomorphism or analyticity, if this function is complex-differentiable in its domain. Formally, for any complex-valued point z 0 , the limit value f (z 0 ) exists, that is,

A.2. Examples
Here we provide two examples to illustrate the holomorphism. The first is the most common linear complex-valued function L(z) = c · z + b, where both c and b are complex-valued constants. So L(z) can also be written as: To prove that the linear complex-valued function L is holomorphic, the following result should hold at every point z: The second example is the quadratic function Q(z) = z 2 . Similarly, Q is holomorphic, according to

A.3. Cauchy-Riemann Condition
Obviously, the limit language is too impractical for identifying a holomorphic function. Fortunately, there is a Cauchy-Riemann condition, which gives a sufficient condition for a complex-valued function to be holomorphic. In detail, f (z) = u(z 1 , z 2 ) + v(z 1 , z 2 )i is differentiable, where u and v both are binary one-valued real functions, if f satisfies the following two terms: • ∂u ∂z 1 , ∂u ∂z 2 , ∂ v ∂z 1 , and ∂v ∂z 2 are continuous in real domain; • The equations ∂u ∂z 1 = ∂v ∂z 2 and ∂u ∂z 2 = − ∂v ∂z 1 hold.

A.4. Adjoint Variables
It is worth noting that the partial derivative matrix of a holomorphic function is anti-symmetric and full-rank. So the following equations hold: which means that the gradients / increments of u, v with respect to z 1 , z 2 are orthogonal, as shown in Figure 7. Furthermore, we can calculate u and v by the integration equations as follows: Obviously, both two integration formulas are path-independent. So we can claim that u(z 1 , z 2 ) is symplectic symmetric to v(z 1 , z 2 ). In addition, if we mastered the value or formulation of u(z 1 , z 2 ), we could easily derive v(z 1 , z 2 ). Therefore, we can notice that M t = v(z 1 , z 2 ) is an adjoint variable of S t = u(z 1 , z 2 ).

A.5. A vivid illustration for adjoint variables
In order to perform the mechanism about adjoint memory thoroughly, we design the following experiments.
Firstly, we generate 3 signals, that is, a cos function with a period of 3, a sin function with a period of 3, and a mixture of the two aforementioned functions over 300 timestamps, as shown in Figure 8(a). We employ only one FT neuron to fit these lines. The results are listed in Figure 8

B. Two Implementation demos of CBP Algorithm
In this section, we provide two detailed implementation demos for CBP algorithm. Specifically, we take one and two layers FTNet as examples.

B.1. One-layer FTNet
One-layer FTNet, or equally the FTO in this paper does not have hidden neurons. Suppose that received signals and output signals are m and n dimension, respectively. Then we can formalize the feed-forward process by: where Y t ∈ R n×1 , W ∈ R n×m , X t ∈ R m×1 , and V ∈ R n×n . In detail, Considering the following cost function E: Then the backpropagation gradients of transmitter concentration matrices W and V through time can be calculated by:

And then we can obtain
Correspondingly, the error correction about imaginary units are provided by After calculation, we can update the coefficient matrices (W, V) according to Finally, correcting the imaginary memory information: M t = σ βŴX t + αVM t−1 .

B.2. Two-layer FTNet
The gradient calculation of two-layer FTNet is homogeneous. Notice that the m-dimensional input signals, l hidden neurons, and n-dimensional output stimuli. Then we can formalize the feed-forward process by: where X t ∈ R m×1 , S t ∈ R l×1 , W ∈ R l×m , V ∈ R l×l , and Y t ∈ R n×1 , P ∈ R n×l , Q ∈ R n×n . In detail, Considering the following cost function E: Then the backpropagation gradients of transmitter concentration matrices (W, V) and (P, Q) through time can be calculated by: Correspondingly, the error correction about imaginary units are provided by: . After calculation, we can update the coefficient matrices (W, V) and (P, Q) according to Finally, correcting the imaginary memory information: