{'fraction_non_alphanumeric': 0.0353527630814398, 'fraction_numerical': 0.005810575246949448, 'mean_word_length': 4.237866410379625, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interpretable Models in ANNs

Artificial neural networks are often very complex and too deep for a human to understand. As a result, they are usually referred to as black boxes. For a lot of real-world problems, the underlying pattern itself is very complicated, such that an analytic solution does not exist. However, in some cases, laws of physics, for example, the pattern can be described by relatively simple mathematical expressions. In that case, we want to get a readable equation rather than a black box. In this paper, we try to find a way to explain a network and extract a human-readable equation that describes the model.


A General View
The term "artificial intelligence" (AI) refers to intelligence demonstrated by man-made machines, in contrast to the natural intelligence of living creatures such as we humans. From the view of computer science, it talks about the study of intelligent agents [1]: A device that is aware of the environment provided and takes actions that try to maximize its chance of successfully achieving some goals. It's often associated with the terms "learning" and "problem-solving".
Since the invention of the computer, many researchers have been exploring the field of AI from the view of computer science via various methods. The field of machine learning (ML) takes the approach of developing computer algorithms that improve through experience [2]. During the last decade, as one of the approaches in ML, artificial neural networks (ANN) have been becoming a more and more popular approach to a lot of real-world complicated problems such as image recognition, decision making, control systems and natural language processing based on complex mathematical models. AlphaGo Zero [3] developed by Google, for instance, is an AI that plays and beats human players in the game of Go. This is mainly due to the increasing computing power of our machines and the availability of a tremendous amount of data. The study of ANN can trace back to [4]. ANN is vaguely inspired by the biological neural networks in our human brain. It's believed that the network "learns" to perform a task by considering previously known examples and concluding patterns.
In modern ANN studies, quite a lot of achievements have been made in the mathematical theories behind the ML principles, and recently, we met some significant milestones in applied fields like self-driving vehicles and natural language processing. However, it is still very common that ANN models are referred to as black boxes, as the network is too complex for a human to interpret. Little effort in the studies of the interpretability of ANN models has been made. As a result, we humans cannot extract any knowledge from the network and do not have much control during the learning process. Although in some areas such as image recognition, the model itself might be fundamentally complex and thus it becomes impossible to describe it simply that a human brain can comprehend, in some other areas, on the other hand, the model is not so hard to describe. Such areas like physics usually have simple laws that can be described using relatively short mathematical expressions. In that case, we would like to find the expression instead of a black box, given that they both work accurately. The main purpose of this paper is to provide a methodology that helps to extract a highly readable mathematical expression from a given ANN that describes a relationship between different variables. Also, as by-products of that, two other problems are considered. First, as we will see later, ANN can only approximate functions on a bounded range of inputs. A model with better extendibility than the original ANN can be found, which should have better accuracy on inputs outside of the range in which the original ANN works. Second, a logical approach to developing the ANN which is different from learning empirical examples is introduced in this paper.
Nowadays, a lot of programming language libraries for ML have been developed by different companies and organizations. One of the most popular ones among them today is TensorFlow [5]. It has outstanding performance and many easy to use application programming interfaces (API), including Keras [6] which is the one used in the experiment part of this paper.
An ANN consists of a number (usually quite large) of neurons. A neuron will perform different tasks in various types of structures of the ANN. Each neuron has an activation function that mimics the process of a biological neuron being "activated". In this paper, we will study the fully connected feed-forward neural network with the rectified linear unit (ReLU) as the activation function.
For the rest of this paper, the abbreviations for fully connected feed-forward neural network, "ANN", "neural network", and "network" will be used interchangeably.

Purpose
The purpose of this paper is to study and examine ReLU networks to provide a high-level description of how they approximate functions and how to interpret them. In the methods section of this paper, we consider viable solutions to the following problems. The first problem is the main focus of this paper, and the latter two are by-products of a solution to the first problem.
• Given a ReLU network that approximates some target function on a bounded range of inputs, find a mathematical expression that approximates the original target function.
• Based on the previous problem, the expression should have better extendibility than the ReLU network, meaning that the expression can be applied to a wider range of inputs with better accuracy.
• Given a mathematical expression, build a ReLU network that approximates it on some bounded range of inputs without training.

Outline
In section 2, we provide the background knowledge of ANN in terms of 3 aspects. How it is formally defined and perform computations internally; how it is trained and learn from experience using a method called backpropagation; and lastly, how it universally approximates a target function while using ReLU as activation functions. Section 3 focuses on a detailed description of the approaches used in this study that tries to address the three problems mentioned above. Section 4 presents the results of the numerical experiments conducted using the methods demonstrated in section 3. Section 5 discusses the results of the previous section. A conclusion is drawn at the end of this section, followed by acknowledgments, the reference list, and the appendix.

What Is ANN
A network is formally defined as the following. An ANN is a mathematical function denoted that takes as input a real-valued vector and computes as output a real-valued vector. The ANN consists of at least two layers sequentially, where the first layer is called the input layer, the last layer is called the output layer, and a layer in the middle, if there is any, is called a hidden layer. Each layer consists of many neurons that perform some computation. In the input layer, the number of neurons is the size of the input vector, with each neuron holds the value of one entry of the vector. The same thing happens for the output layer. Starting from the second layer, each neuron takes as input all the output values of neurons from the last layer and then computes a weighted sum of those inputs. This sum is to be added with a bias term and then sent into an activation function. Finally, the output of the activation function will be the output of this neuron. Each layer has its activation function and each neuron has its weights and bias. Using the matrix and vector notation (the arrow notation will be omitted for the rest of this paper when it's clear from context for conciseness), a one hidden layer ANN can be expressed as follows. Let ⃗ be the input vector. Let 1 and 2 be the matrix representing the weights of the hidden layer and output layer, respectively, where each row is the weights for one neuron. Let 1 ⃗⃗⃗⃗ and 2 ⃗⃗⃗⃗⃗ be the vector representing the biases for those two layers, respectively, where each entry is the bias for one neuron. Finally, let 1 and 2 be activation functions.
Depending on how the activation functions are defined, the network can perform different tasks. For example, if the task is to compute some kind of probability, 2 can be the sigmoid function ( ) = 1 1+ − , as the output is in the range (0,1). If the task is regression, 2 can be the identity function. For this paper, we consider only regression problems (function approximation). ReLU has become a popular alternative to the sigmoid function recently, due to its computational simplicity and other nice properties [7]. The ReLU function is formally defined as We will focus on this specific activation function for the latter parts.

Backpropagation
Because the detailed training process of the neural network is not part of the main focus of this paper and will not be mentioned again for the rest of this paper, here we only provide a high-level view of how the training is performed. The network learns the model using a method called backpropagation [8]. Initially, the weights and biases in an ANN are randomly assigned. To train the network is to provide examples and try to adjust the weights and biases in the network such that the new output of the ANN is closer to the target value according to the example. Given an example input-output pair, we measure how close the computed output and the target output is through something called the loss function, also referred to as the error function. For example, a commonly used loss function is the squared loss. Let be the true vector value according to the example, and let ̂ be the predicted vector by the ANN. It is formally defined as ( ,̂) = ( −̂) 2 (3) We then use the gradient descent method (originally proposed by A. L. Cauchy in 1847) to minimize the loss function. Thus, the network is refined and will hopefully generate a better output that is closer to the true value next time. The high-level idea is that if we consider the function of the error measured on that specific example in terms of the weights and biases, there is a direction in which the error is dropping most rapidly according to the gradient. We adjust the weights and biases in that direction, so the error is reduced on that specific example. Note that this is essentially a mathematical optimization problem. It is also possible (normally better) to consider the average error measured among multiple examples [9]. Such a set of examples is called a mini-batch. As we train the network on a set of examples called the training sample set, the empirical error will decline. In the end, we are expected to get a good ANN that grasps some kind of pattern between the input and output.

A View of Function Approximation
The universal approximation theorem [10] states that on any compact subset of ℝ , and for any function that is continuous in that set, there is a feed-forward network with a single hidden layer that can approximate the function as close as we want. under mild assumptions on the activation function. The universal approximation theorem for width-unbounded networks can be expressed as follows: For all compact subset ⊊ ℝ , for all continuous function : → ℝ, for all > 0, there exists one hidden layer fully connected network with some activation function, such that the function : ℝ → ℝ represented by this network satisfies [11] proved that this holds for networks that use ReLU as activation functions. It's been shown that a feed-forward artificial neural network is essentially a linear spline [12,13]. We provide a high-level explanation in this paper. Say you have a network using ReLU as activation functions. The network that takes as input ∈ ℝ , an n-dimensional real-valued vector, and computes as output = ( ) ∈ ℝ, a real number. Recall equation (2), where ReLU is defined as max({0, }). We say the neuron is activated when the output is greater than 0. We can see when the ReLU is activated, it is simply the identity function ( ) = . In other words, it describes a straight line on the graph. This is the key idea of how a ReLU network approximate functions.
Consider a specific input vector , let the network compute y as output, then we remove all the neurons in the hidden layers that are not activated. When a neuron is not activated it outputs 0, thus it contributes nothing to the next layer, no matter what its outgoing weights are, so the output of the network is still y. Now we remove the activation functions, the remaining network still outputs y since ReLU is just the identity function when it is activated and removing it changes nothing. Now we can see y is computed as an affine transformation of its last layer's outputs, and so is every neuron in its last layer. Since an affine transformation of another affine transformation of a set of real values is still just an affine transformation of values in the set , ultimately the output of the neural network is an affine transformation of the entries of the input , denoted with ⋅ + . As long as the set of activated neurons remains the same, the network will keep computing as output ( ) = ⋅ + .
As we change the input , some new neurons might be activated. We know the process of changing the set of activated neurons is discrete, thus there must be a range of input, such that the original network will keep computing as output (̃) = ⋅̃+ , for all ̃ that is in that range (i.e. close enough to ). The idea is that the network is just using a local linear approximation to predict the target function near . Thus, it is essentially a linear spline. More formally, we have, For all ∈ ℝ , there exists a ∈ ℝ and a compact subset where { } ⊊ ⊊ ℝ , such that for all ̃∈ , This tells us a ReLU network is drawing the target function using tiny flat pieces. The accuracy, of course, depends on the number of knots and the distances between them (i.e. step size). This explains why ReLU networks can approximate any function that is continuous in the compact subset where we are making our approximation. As the function is continuous in the compact subset, the change in the function's output can get arbitrarily small if the change in the input is small. More formally, a function is continuous in an input domain means ∀ 0 ∈ , ∀ > 0, ∃ > 0, ∀ ∈ : Thus, using small enough linear pieces, the error can be arbitrarily small for any continuous target function.

From Slope to Derivative
Now we know how ReLU networks approximate functions. We then use this property of ReLU to construct an expression that approximates the target function. For simplicity, let's consider a simple network (not necessarily using ReLU) where the function represented by it is : ℝ → ℝ which approximates the target function on some closed interval [ , ] . Assume satisfies condition (6) Furthermore, if is real analytic, then locally Locally means that the equality holds on some interval ( , ) that includes 0 if there is no discontinuity on ( , ). Now we show how to build the function ̂: [ , ] → ℝ, which tries to recover by using a smooth approximation of . This is easy for ANNs that use ReLU as activation functions, which are essentially linear splines. There are infinitely many functions that can have the linear spline , so let's pick one that is easy to work with, namely a real analytic function ̂. The final expression that approximates the model will be stated as a finite number of terms of its Taylor series.
To get a Taylor series of ̂, we need to get the derivatives of it. Once we have different orders of its derivatives, we can just pick a point 0 and plug values in to get the Taylor The average derivative of ̂ over [ , ] is simply the slope of the line segment defined by on [ , ]. is almost everywhere differentiable, except at the knots. ′ ( ) is also the slope of the line segment at . Recall that ̂≈ ( ) with error bound . If is small enough, we have ′ ( ) ≈̂′( ) on [ , ]. Note this property holds because of the nice property of linear spline. It may not hold for activation functions other than ReLU. In other words, the ′ is the slope which is also the average values of ̂′ which can be a good approximation to ̂′ itself. The slope is easily computed if we look at the weights inside the neural network. Let the output of an activated neuron be (ℎ ⃗⃗ ) = ⃗⃗⃗ ⋅ ℎ ⃗⃗ + where ⃗⃗⃗ and are the weights and bias for that neuron and ℎ ⃗⃗ is the vector of outputs from its last layer and let ℎ ⃗⃗ denotes its -th entry. The deactivated neurons just output 0. To compute ′ ( ) is to compute ( ) where is the neuron in the output layer. The derivative of the output of a neuron can be computed recursively using the chain rule.
Since ̂ is real analytic, ̂′ is real analytic, too. Thus, there is another ANN learns a good approximation to it. To get training examples for our new ANN that will learn ̂′ , we apply this method of computing slopes to different line segments of the linear spline. Then, we get a table of discrete approximate values of ̂′ ( ) on [ , ], which can be used as the training sample set for the new ANN. We can obtain as many samples as we need, so the new ANN can fit the derivative of the previous ANN arbitrarily well. Then, we can repeat this process to get an approximation to ̂′ ′ ,̂′ ′′ , and so on. Once we have those, all we need to do is to pick a value 0 and plug it into the neural networks to get different orders of derivatives. Let ( ) be the function represented by the ANN that learns ̂( ) , so (0) = learns ̂. Plug those values into (8). We get Theoretically, if the neural networks are accurate, the infinite series should be the same no matter which point we chose as our 0 , but with some inevitable error in the networks, it is a good idea that we pick several 0 , get a Taylor series from each of them, and take the average. The average error must be no greater than the greatest error among them, so it has better accuracy. Let be the Taylor series obtained from the -th choice of 0 . We use the following average as our Taylor series.
Combining expression (11) with expression (12) we have Practically speaking, we have to stop the algorithm at some point. If we are lucky, in the case of the target function is a polynomial, we get a finite series and the algorithm stops as we get a constant derivative of 0. More commonly, however, if the target function is not a simple polynomial, we end up with an infinite Taylor series. Based on the precision we desire depending on the practical situation of the real-world problem, we need to chop off the series at some point and obtain a finite polynomial. As we chop off the series at some point, the series is only a local approximation centred at our choice of 0 . In that case, take an average of several Taylor series can lead to a worse result, because each of them is for a different 0 . We should just choose the point in the range where we want to approximate. Let be the order of Taylor series we want. We compute the following as our final approximation to the refined version of ( ), which goes back to approximating the original target function .
Note that it is also possible to replace ( ) ( 0 ) with ( −1) ′ ( 0 ), but the result is not as good. The details will be explained in section 3.4.
The algorithm is demonstrated in a high-level sense in the following pseudocode. Note that "//" is followed by a line of comments.

From Derivative to Gradient
Using the same idea, one can easily adapt the method to higher dimensions. For convenience, let's denote the functions with multiple inputs with functions that take a vector as input. The arrow notation is used to avoid the confusion in subscripts. Let the function : ℝ → ℝ be an analytic function, ⃗ be the input vector and 0 ⃗⃗⃗⃗⃗ be a specific input. Let the notation ⃗ denote the -th entry of ⃗. Consider the Taylor series in -dimensional space: Applying the multi-index notation, the series can be written as a more compact form, Here is an example of a second-order Taylor series of a function with 2-dimensional input. Let ∇ 2 be the Hessian matrix notation that represents second-order partial derivatives.
The partial derivatives are also easily computed looking at the weights inside the ReLU network, as the function is made of tiny flat pieces. Equation (10) is now adapted to the following form Based on the algorithm for 1D Taylor series in section 3.1, the general algorithm is described in a higher-level language in the following pseudocode. It is similar to the one illustrated in section 3.1, except that here it computes partial derivatives needed for the Taylor series instead of derivatives. The interval we were working on, [ , ], is expanded to a compact subset of ℝ naturally. The sequence of ReLU networks is trained in a breadth-first search (BFS) fashion, where the is the root node with children ⃗ 1 , … , ⃗ . A nice property of partial derivatives and save us some effort. Since the order in which we take partial derivatives does not matter for analytic functions (e.g. = ), we only need to compute some branches. while ≤ do: Although it was not the main goal of this study that this method, as a side product, provides a possible way of obtaining a model with better extendibility than the original ANN. As the universal approximation theorem described in inequality (4), the ANN is only expected to approximate the target function on a closed interval. Any ReLU network cannot realize the simple polynomial ( ) = 2 , as the linear spline eventually only grows linearly. With the method introduced in this section, there is a chance to recover the target function on a universal scale or at least get better accuracy, if the target function is a finite polynomial. Later we will see similar examples in the experiments section.

Determine the Order
One problem remains. How do we select the order of Taylor series we want? Unfortunately, this is a hopeless problem, unless we get unlimited oracle access (in the sense of a black box) to the target function. Because we only have sample data in a limited range, no numerical analysis method can tell the difference between some functions without perfect precision.
This is exactly what happened hundreds of years ago when Galileo Galilei (oddly enough, most people only know his first name) came up with an equation for velocity adding, = 1 + 2 . However, it is not until centuries later that A. Einstein proved that is wrong according to his special theory of relativity. They are two of the greatest scientists of all time, so what went wrong? According to the special theory of relativity, a better (maybe correct) model for velocity along one axis is = 1 + 2 1+ 1 2 2 ⁄ where ≈ 3 × 10 8 meters per second (the speed of light). We can see that two models are very close under low speed as the denominator 1 + 1 2 2 ⁄ ≈ 1. The two are identical in cases where we use a measuring device or a computer with imperfect precision. If we cannot do experiments under very high speed, we will never realize the problem.
However, there is still something we can do. Suppose we have an ANN that learned the target function, then we can use the following trick. Fix all input variables except for one of them. Let's call it . Now let the network compute different outputs for different values of . We then obtain a graph of the target function's 2D projection, denoted ( ). Many numerical analysis methods can be applied. We could, for example, use the finite-difference method (first proposed by L. Euler in 1768) to guess the degree of in the polynomial. The whole idea of the Taylor approximation in section 3.1 and 3.2 is a finite-difference method. Or if we get access to very big values of , then we can draw the graph of where is the degree of ( ). If the coefficient of the term with the highest degree is negative and the function seems to decrease as → ∞, just try to lift the function. This will also work the same way.
We do that for each variable to get the overall degree of the polynomial. If we are lucky, the underlying model is indeed a polynomial then we might get the correct degree. Otherwise, choosing a high enough order is just a matter of accuracy. What do statisticians say? "All models are wrong, but some are useful".

Technical Notes
During the experiments, it has been found that the predicted values near the boundary of the training data set, unsurprisingly, have lower accuracy. A useful trick for training the ANNs is to sample the derivatives in a smaller area than where the previous training sample set was from. Thus, we avoid the bad training values of derivative on the edges. Another noticeable point is that when the underlying model is given by a finite polynomial, the Taylor series will come to an end when it gets derivatives of constant 0. We should choose the precision based on training error (empirical loss). For example, if the training error is 0.01, then it is meaningless to compute the derivatives to the third decimal place. This helps a lot when it comes to terminating the Taylor series when it is given by a finite polynomial as the ending derivatives will be very close to 0.
As mentioned after equation (14). We can use either the calculated slope or the learned derivative as the training sample. During the experiments, it's found that the learned derivative might be a better choice. ReLU networks are good at fitting a target function, but the derivatives are not so accurate. In the figure below, the derivative (slopes) of the ANN is shown in blue, and the target function's actual derivative is shown in orange. We can see that the overall trend is not bad, but not the individual values. Letting another ANN to learn it with some regularization (limiting the model complexity) can address this problem.

Reverse Engineering
Now we know how to extract an expression from a ReLU network. Theoretically although very tediously, we can do the reverse. Given a mathematical expression as the continuous target function, we can manually set weights in the ReLU network to make it fit. It is shown in section 2.3 that ReLU networks are line splines. To make the line spline of a given function, we simply put little linear pieces according to the function's derivative. The idea is to have neurons that check the range of input, and a neuron for a linear summation over different dimensions. Overall, the network has two hidden layers. This is a similar idea to [14] where the sigmoid function as the activation function is considered, but the derivatives do not match. In this section, we provide a method that not only matches the function values, but also the derivatives.
For example, if we want to draw the function ( ) = 2 , we first calculate its derivative, namely 2 . Now, say we want to use three line segments on the interval (1,2], (2,3], (3,4], respectively. Using the middle value of each interval to calculate the derivative, we need lines of slope 3, 5, 7, respectively. We build neurons in the first hidden layer that work in triples to perform the task of identifying the range of inputs and build a neuron in the second hidden layer to sum the triple up. For convenience, let's define −∞ to be negative infinity (or just a negative number that's small enough to work). Define (−∞)0 = 0. Consider the incoming weights for 3 neurons in the first hidden layer. Set the weights to be -1, 1 and 1, respectively. Set the biases to be 1, -1 and -2, respectively. Now we can see that, the first neuron will be activated when < 1; the second one will be activated when > 1; and the last one will be activated when > 2. Set the outgoing weights of them to be −∞, 3 and −∞, respectively. When ∈ (1,2], only the second one is activated, and the triple together contribute 3( − 1) to the overall output. When < 1 or > 2, however, one of the neurons with outgoing weight −∞ is turned on and dominates the sum of the triple, so the triple together outputs a sum of −∞. Thus, the summing neuron in the next layer will output 0 after ReLU. As a result, this unit of four neurons will put a line segment of slope 3 on the interval [1,2]. The next group of four is constructed for the next interval the same way. Then we successfully obtain a line spline based on derivatives. The last thing to do is to set the bias for the neurons in the second layer. This will have the effect of moving the line segment up and down. We just put it wherever fits the best.
Note that this method is essentially a constructive version of the universal approximation theorem for two hidden layers ReLU networks. One can easily generalize this method to higher dimensions. The triples will be updated to sets of triples that control each dimension. Note that the −∞ weights and the summing neuron in the second layer together work essentially as a logical AND gate. As long as at least one of the inputs is no in the desired range the overall output will be 0. Also, if we accept the definition of −∞, then the ReLU network will have the potential to fit piecewise functions (but not general discontinuous functions), because the ANN is now discontinuous.

A More General Sense
It is possible to solve (but not practically) any finite problem with the Taylor series. Let's formally define what a problem is and what it means to solve a problem. A problem is a function : → , with question space and answer space . A solution to the problem is a function : → such that ∀ ∈ : ( ) = ( ). Solving the problem means finding a computable solution. A problem is called finite when | | < ∞, | | < ∞. Consider a finite problem : → , let | | = and | | = . We can encode every input and output values using real numbers, as they are finite. We then draw the mapping from real values to their corresponding real values. That gives us a set of points {( 1 , ( 1 )), … , ( , ( ))}. By the interpolation theorem (as a result of the fundamental theorem of algebra), there is a unique polynomial of degree − 1 that passes through all the points, denoted ′ . Thus, the function : → , ( ) = ( ′ ( ( ))) where and are the encoding and decoding for questions and answers, respectively, is a solution to the problem. ′ is a polynomial thus it is its own Taylor series.

Experiments
In each of the sections below, a target function is to be recovered using the algorithm introduced in section 3. Given that I do not have a powerful computing device, some simple functions are tested. All the underlying models tested are the ones that are impossible for ReLU networks to universally realize. Since one-hidden-layer ANNs already are universal function approximators, I am using only ReLU networks with a single hidden layer. It is easier to compute the gradient. For all the figures, blue represents the output values of the ANN and orange is the desired true value. The gap between the predicted values and the actual ones might appear larger than actual. Notice the scale of the axis.

A Simple Function
( ) = 0.5 2 − 2 + 1 To make the task harder, the training sample data points are not evenly spaced. Results:

Conclusion
For the first target function, the ANN that directly learns the target function was a bit off in the middle due to a lack of sample data. The result was pretty good. It not only successfully recovered the order of the function, also got a high accuracy on the original interval. The second test result was also ideal. One of the local approximations even perfectly recovers the model. The coefficients of the last two terms in the calculated model were quite low. One may confirm that we can remove those two with further experiments. Additionally, the resulting polynomial has better accuracy than the original ReLU network on the sample interval. Overall, the goals of this paper were successfully met. Python 3 is a popular programming language that has a large amount of ML libraries. It is known for its applications in scientific researches and the high readability of the code.

A.2. NumPy 1.18.1
NumPy is a widely used library for numerical computing. It is also a dependency for a lot of ML libraries such as TensorFlow.

A.3. TensorFlow 2.1.0
TensorFlow is the most popular ML library in the world today, with great performance and flexibility. It was first released by Google Brain Team in November 2015.

A.4. Keras 2.3.1
Keras is a library that provides an easy to use tool kit for ML. It can also run on top of other ML libraries like TensorFlow. In the experiments, Keras is used as a high-level API of TensorFlow.

A.5. Matplotlib 3.2.0
Matplotlib is a very powerful library for drawing figures.

A.6. Python 3 code of the ANN structure used in the experiments
It includes the function that computes the gradient. self.nnet.save(name + ".h5")