{'fraction_non_alphanumeric': 0.044273294273294274, 'fraction_numerical': 0.009497259497259497, 'mean_word_length': 4.172287390029325, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Neural Transfer Function for a Smooth and Differentiable Transition Between Additive and Multiplicative Interactions

Existing approaches to combine both additive and multiplicative neural units either use a fixed assignment of operations or require discrete optimization to determine what function a neuron should perform. This leads either to an inefficient distribution of computational resources or an extensive increase in the computational complexity of the training procedure. We present a novel, parameterizable transfer function based on the mathematical concept of non-integer functional iteration that allows the operation each neuron performs to be smoothly and, most importantly, differentiablely adjusted between addition and multiplication. This allows the decision between addition and multiplication to be integrated into the standard backpropagation training procedure.


Introduction
In commonplace artificial neural networks (ANNs) the value of a neuron is given by a weighted sum of its inputs propagated through a non-linear transfer function. For illustration let us consider a simple neural network with multidimensional input and multivariate output. The input layer should be called x and the outputs y. Then the value of neuron y i is (1) The typical choice for the transfer function σ(t) is the sigmoid function σ(t) = 1/(1 + e −t ) or an approximation thereof. Matrix multiplication is used to jointly compute the values of all neurons in one layer more efficiently; we have where the transfer function is applied element-wise, σ i (t) = σ(t i ). In the context of this paper we will call such networks additive ANNs. (Hornik et al., 1989) showed that additive ANNs with at least one hidden layer and a sigmoidal transfer function are able to approximate any function arbitrarily well given a sufficient number of hidden units. Even though an additive ANN is an universal function approximator, there is no guarantee that it can approximate a function efficiently. If the architecture is not a good match for a particular problem, a very large number of neurons is required to obtain acceptable results. (Durbin & Rumelhart, 1989) proposed an alternative neural unit in which the weighted summation is replaced by a product, where each input is raised to a power determined by its corresponding weight. The value of such a product unit is given by Using laws of the exponential function this can be written as y i = σ[exp( j W ij log x j )] and thus the values of a layer can also be computed efficiently using matrix multiplication, i.e.
where exp and log are taken element-wise. Since in general the incoming values x can be negative, the complex exponential and logarithm are used. Often no non-linearity is applied to the output of the product unit.

Hybrid summation-multiplication networks
Both types of neurons can be combined in a hybrid summation-multiplication network. whether a specific neuron should be an additive or multiplicative unit to obtain the best results. A simple solution is to stack alternating layers of additive and product units, optionally with additional connections that skip over a product layer, so that each additive layer receives inputs from both the product layer and the additive layer beneath it. The drawback of this approach is that the resulting uniform distribution of product units will hardly be ideal.
A more adaptive approach is to learn the function of each neural unit from provided training data. However, since addition and multiplication are different operations, until now there was no obvious way to determine the best operation during training of the network using standard neural network optimization methods such as backpropagation. An iterative algorithm to determine the optimal allocation could have the following structure: For initialization randomly choose the operation each neuron performs. Train the network by minimizing the error function and then evaluate its performance on a validation set. Based on the performance determine a new allocation using a discrete optimization algorithm (such as particle swarm optimization or genetic algorithms). Iterate the process until satisfactory performance is achieved. The drawback of this method is its computational complexity; to evaluate one allocation of operations the whole network must be trained, which takes from minutes to hours for moderately sized problems.
Here we propose an alternative approach, where the distinction between additive and multiplicative neurons is not discrete but continuous and differentiable.
Hence the optimal distribution of additive and multiplicative units can be determined during standard gradient-based optimization. Our approach is organized as follows: First, we introduce non-integer iterates of the exponential function in the real and complex domains. We then use these iterates to smoothly interpolate between addition (1) and multiplication (3). Finally, we show how this interpolation can be integrated and implemented in neural networks.

Functional iteration
Let f : C → C be an invertible function. For n ∈ Z we write f (n) for the n-times iterated application of f , Further let f (−n) = (f −1 ) (n) where f −1 denotes the inverse of f . We set f (0) (z) = z to be the identity function. It can be easily verified that functional iteration with respect to the composition operator, i.e.
for n, m ∈ Z, forms an Abelian group.
Equation (5) cannot be used to define functional iteration for non-integer n. Thus, in order to calculate non-integer iterations of function, we have to find an alternative definition. The sought generalization should also extend the additive property (6) of the composition operation to non-integer n, m ∈ R.

Abel's functional equation
Consider the following functional equation given by (Abel, 1826), with constant β ∈ C. We are concerned with f (x) = exp(x). A continuously differentiable solution for β = 1 and x ∈ R is given by Note that for x < 0 we have k = −1 and thus ψ is well defined on whole R. The function is shown in Fig. 1. Since ψ : R → (−1, ∞) is strictly increasing, the inverse ψ −1 : (−1, ∞) → R exists and is given by with k ∈ N s.t. 0 ≤ exp (k) (ψ) < 1. For practical reasons we set ψ −1 (ψ) = −∞ for ψ ≤ −1. The derivative of ψ is given by with k ∈ N s.t. 0 ≤ log (k) (x) < 1 and the derivative of its inverse is

Non-integer iterates using Abel's equation
By inspection of Abel's equation (7), we see that the nth iterate of the exponential function can be written as While this equation is equivalent to (5) for integer n, we are now also free to choose n ∈ R and thus (11) can be seen as a generalization of functional iteration to non-integer iterates. It can easily be verified that the composition property (6) holds. Hence we can understand the function ϕ(x) = exp (1/2) (x) as the function that gives the exponential function when applied  Figure 1. A continuously differentiable solution ψ(x) to Abel's equation (7) for the exponential function in the real domain. Figure 2. Iterates of the exponential function exp (n) (x) for n ∈ {−1, −0.9, . . . , 0, . . . , 0.9, 1} obtained using the solution (11) of Abel's equation.
to itself. ϕ is called the functional square root of exp and we have ϕ(ϕ(x)) = exp(x) for all x ∈ R. Likewise exp (1/N ) is the function that gives the exponential function when iterated N times.
Since n is a continuous parameter in definition (11) we can take the derivative of exp with respect to its argument as well as n. They are given by Thus (8) provides a method to interpolate between the exponential function, the identity function and the logarithm in a continuous and differentiable way.

Schröder's functional equation
Motivated by the necessity to evaluate the logarithm for negative arguments, we derive a solution of Abel's equation for the complex exponential function. Applying the substitution in Abel's equation (7) gives a functional equation first examined by (Schröder, 1870), with constant γ ∈ C. As before we are interested in solutions of this equation for f (x) = exp(x); we have but now we are considering the complex exp : C → C.
The complex exponential function is not injective, since exp(z + 2πni) = exp(z) n ∈ Z .
Thus the imaginary part of the codomain of its inverse, i.e. the complex logarithm, must be restricted to an interval of size 2π. Here we define log : C → {z ∈ C : β ≤ Im z < β + 2π} with β ∈ R. For now let us consider the principal branch of the logarithm, that is To derive a solution, we examine the behavior of exp around one of its fixed points. A fixed point of a function f is a point c with the property that f (c) = c.
The exponential function has an infinite number of fixed points. Here we select the fixed point closest to the real axis in the upper complex half plane. Since log is a contraction mapping, according to the Banach fixed-point theorem (Khamsi & Kirk, 2001) the fixed point of exp can be found by starting at an arbitrary point z ∈ C with Im z ≥ 0 and repetitively applying the logarithm until convergence. Numerically we find Close enough to c the exponential function behaves like an affine map. To show this, let z = z − c and consider Here we used the Taylor expansion of the exponential function, exp(z ) = 1 + z + O(z 2 ). Thus for any point z in a circle of radius r 0 around c, we have By substituting this approximation into (14) it becomes apparent that a solution to Schröder's equation around c is given by where we have set γ = c.
We will now compute the continuation of the solution to points outside the circle around c. From (14) we obtain χ(z) = c χ(log(z)) .
If for a point z ∈ C repeated application of the logarithm leads to a point inside the circle of radius r 0 around c, we can obtain the function value of χ(z) from (16) via iterated application of (17). In the next section it will be shown that this is indeed the case for nearly every z ∈ C. Hence the solution to Schröder's equation is given by with k = min k ∈N k s.t. | log (k ) (z) − c| ≤ r 0 . Solving for z gives with k = min k ∈N k s.t. |c −k χ| ≤ r 0 . Obviously we have χ −1 (χ(z)) = z for all z ∈ C. However χ(χ −1 (ξ)) = ξ only holds if Im (c −k ξ +c) ∈ [β, β +2π).
The derivative of χ is given by 2.3.1. The solution χ is defined on almost C The principal branch of the logarithm, i.e. restricting its imaginary part to the interval [−π, π), has the drawback that iterated application of log starting from a point on the lower complex half-plane will converge to the complex conjugate c instead of c. Thus χ(z) would be undefined for Im z < 0.
To avoid this problem, we use the branch defined by log : C → {z ∈ C : β ≤ Im z < β + 2π} with −1 < β < 0. Using such a branch the series z n , where converges to c, provided that there is no n such that z n = 0. Thus χ is defined on C \ D where D = {0, e, e e , e e e , . . . }.
On the other hand, if Im z n < 0 and Re z n < 0, then Im log z n > 0 and z n converges as above. Finally, if Im z n < 0 and Re z n ≥ 0, then, using −1 < β, we have Re z n+1 ≤ | log z n | ≤ 1 + log(Re z n ) < Re z n and thus at some element n in the series we will have Re z n < 1 which leads to Re z n +1 < 0. Re χ(z) Im χ(z) Figure 5. Structure of the function χ(z) and calculation of exp (n) (1 + πi) for n ∈ [0, 1] in z-space (left) and χ-space (right). The uniform grid with the cross at the origin is mapped using χ(z). The points 1 + πi and ξ = χ(1 + πi) are shown as hollow magenta circles. The black line shows χ −1 (c n ξ) and c n ξ for n ∈ [0, 1]. The blue and purple points are placed at n = 1/2 and n = 1 respectively.

Non-integer iterates using Schröder's equation
Repetitive application of Schröder's equation (13) on an iterated function (5) leads to Thus the nth iterate of the exponential function on the whole complex plane is given by where χ(z) and χ −1 (z) are given by (18) and (19) respectively. Since χ is injective we can think of it as a mapping from the complex plane, called z-plane, to another complex plane, called χ-plane. By (22) the operation of calculating the exponential of a number y in the z-plane corresponds to complex multiplication by factor c of χ(y) in the χ-plane. This is illustrated in Fig. 5. Samples from exp (n) are shown in Fig. 7.
While the definition for exp (n) given by (22) can be evaluated on the whole complex plane C, it only has meaning as a non-integer iterate of exp, if composition exp (n) [exp m (z)] = exp (n+m) (z) holds. Since this requires that χ(χ −1 (ξ)) = ξ, let us define the sets Then, for z ∈ E, n ∈ R and m ∈ [−1, 1], the composition of the iterated exponential function is given by and the composition property is satisfied. The subset E of the complex plane where composition of exp (n) for non-integer n is shown in figure 6.
The derivatives of exp (n) defined using Schröder's equation are given by Hence we defined the continuously differentiable function exp (n) : C \ D → C on almost the whole complex plane and showed that it has the meaning of a noninteger iterate of exp on the subset E.

Interpolation between addition and multiplication
Using fundamental properties of the exponential function we can write every multiplication of two numbers x, y ∈ R as xy = exp(log x + log y) = exp(exp −1 x + exp −1 y) .
We define the operator ⊕ n for x, y ∈ R and n ∈ R as Note that we have x ⊕ 0 y = x + y and x ⊕ 1 y = xy. Thus for 0 < n < 1 the above operator continuously interpolates between the elementary operations of addition and multiplication. We will refer to ⊕ n as the "addiplication operator". Analogous to the n-ary sum and product we will employ the following notation for  Figure 8. The interpolation between addition and multiplication using (24) with x = 2 and y = 5. The iterates of the exponential function are either calculated using Abel's equation (pink) or Schröder's equation (magenta). In both cases the interpolated values exceed the range between 2 + 5 = 7 and 2 · 5 = 10 and therefore a local maximum exists.
the n-ary addiplication operator, The derivative of the addiplication operator w.r.t. its operands and the interpolation parameter n are calculated using the chain rule. Using the shorthand For positive arguments x, y > 0 we can use the iterates of exp based either on the solution of Abel's equation (11) or Schröder's equation (22). However, if we also want to deal with negative arguments, we must use iterates of exp based on Schröder's equation (22), since the real logarithm is only defined for positive arguments. From the exemplary addiplication shown in Fig. 8 we can see that the interpolations produced by these two methods are not monotonic functions w.r.t. the interpolation parameter n. In both cases local maxima exist; however interpolation based on Schröder's equation has higher extrema in this case and also in general (personal experiments). It is well known, that the existence of local extrema can pose a problem for gradient-based optimizers.

Neurons that can add or multiply
We propose two methods to construct neural nets that have units the operation of which can be adjusted us-ing a continuous parameter.
The straightforward approach is to use neurons that use addiplication instead of summation, i.e. the value of neuron y i is given by For n i = 0 the neuron behaves like an additive neuron and for n i = 1 it computes the product of its inputs. Because we sum over exp (−ni) (x j ) which has dependence on the parameter n i of neuron y i , this calculation corresponds to a network in which each neuron in layer x has separate outputs for each neuron in the following layer y; see Fig. 9a. Compared to conventional neural nets this architecture has only one additional real-valued parameter per neuron (n i ) but also poses a significant increase in computational complexity due to the necessity of separate outputs. Since exp (−ni) (x j ) is complex it might be sensible (but is not required) to allow a complex weight matrix W ij .
The computational complexity of separate output units can be avoided by calculating the value of a neuron according to This corresponds to the architecture shown in Fig. 9b. The interpolation parameter n i has been split into a pre-transfer-function partn yi and a post-transferfunction partñ xj . Sincen yi andñ xj are not tied together, the network is free to implement arbitrary combinations of iterates of the exponential function. Addiplication occurs as the special casen yi = −ñ xj . Compared to conventional neural nets each neuron has two additional parameters, namelyn yi andñ xj ; however the asymptotic computational complexity of the network is unchanged. In fact, this architecture corresponds to a conventional, additive neural net, as defined by (1), with a neuron-dependent, parameterizable transfer function. For neuron z i the transfer function given by Consequently, implementation in existing neural network frameworks is possible by replacing the standard sigmoidal transfer function with this function and optionally using a complex weight matrix.  (27). We have yi = σ(ŷi) and the subunits computexij = exp (−n i ) (xj) and yi = exp (n i ) j Wijxij . The weights between the subunits are shared. (b) Neuron yi calculates its value according to (28). We have yi = σ(ŷi) and the subunits computẽ xj = exp (−ñ j ) (xj) andŷi = exp (n i ) j Wijxj .

Variable pattern shift
Consider the dynamic pattern shift task shown in Fig. 10a: The input consists of a binary vector x of N elements and an integer m ∈ {0, 1, . . . , N − 1}. The desired output y is x circularly shifted by m elements to the right, where x is indexed modulo N , i.e. x n−m rolls over to the right if n − m ≤ 0.
A method to architecturally efficiently implement this task in a neural architecture is based on the shift theorem of the discrete Fourier transform (DFT) (Brigham, 1988). Let F({x n }) k denote the kth element of the DFT of x = {x 0 , . . . , x N −1 }. By definition we have x n e −2πi kn N and its inverse is given by The shift theorem states that a shift by m elements in the time domain is equivalent to a multiplication by factor e −2πikm/N in the frequency domain, Hence the shifted pattern can be calculated using Using the above definitions its vth component is given by Figure 10. (a) Variable pattern shift problem. Given a random binary pattern x ∈ R N and an integer m ∈ {0, 1, . . . , N −1} presented in one-hot encoding, the learner should output the pattern x circularly shifted to the right by m grid cells. (b) A neural net with two hidden layers that can solve this problem by employing the Fourier shift theorem. The first hidden layer is additive, the second is multiplicative; the output layer is additive. All neurons use linear transfer functions. The first hidden layer computes the DFT of the input pattern x and shift mount s. The second hidden layer applies the Fourier shift theorem and the output layer computes the inverse DFT of the shifted pattern.
If we encode the shift amount m as a one-hot vector s of length N , i.e. s j = 1 if j = m else s j = 0, we can further rewrite this as x n e −2πi kn N .
This corresponds to a neural network with two hidden layers (one additive, one multiplicative) and an additive output layer as shown in Fig. 10b. The optimal weights of this network are given by the corresponding coefficients from (30).
From this example we see that having the ability to automatically determine the function of each neuron is crucial to learn neural nets that are able to solve complex problems.

Conclusion and future work
We proposed one method to continuously and differentiably interpolate between addition and multiplication and showed how it can be integrated into neural networks by replacing the standard sigmoidal transfer function with a parameterizable transfer function. In this paper we presented the mathematical formulation of these concepts and showed how to integrate them into neural networks.
We will perform simulations to see how these neural nets behave given real-world problems and how to train them most efficiently.
While working on this theory, we already have two possible improvements in mind: 1. Our interpolation technique is based on noninteger iterates of the exponential function calculated using Abel's and Schröder's functional equations. We chose this method for our first explorations because it has the mathematically sound property that the calculated iterates form an Abelian group under functional composition. However it results in a non-monotonic interpolation between addition and multiplication which may lead to a challenging optimization landscape during training. Therefore we will try to find more ad-hoc interpolations with a monotonic transition between addition and multiplication.
2. Our method introduces one or two additional realvalued parameters per neuron for the transfer function. Using a suitable fixed transfer function might allow to absorb these parameters back into the bias.
While the specific implementation details proposed in this work may have their drawbacks, we believe that neurons which can implement operations beyond addition are a key to new areas of application for neural computation.