{'fraction_non_alphanumeric': 0.05656706045865184, 'fraction_numerical': 0.012369701181375956, 'mean_word_length': 3.945704467353952, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On a partial differential equation based neural network

A continuous formulation of a neural network based on a partial differential equation (PDE) is proposed. By assuming binary classification with cross-entropy, we formulate the neural network as an optimal control problem of a PDE. The existence of the optimal solution is theoretically proved. By finding the solution and optimal control in Hilbert spaces, we provide the explicit representation of the Fréchet derivative of the cost function at the solution. We also propose a gradient-descent-based algorithm that delivers a (sub-) optimal control.


Introduction
Since it was first proposed, ODENet [1] has attracted significant interest in the research field, although it can be further improved.
A similar idea can be found in [2], in which they formulated the neural network as a form of an ODE: x t = f (x t , θ t ), where x t is the continuous limit of the states of nodes at time t, f (·) is a certain nonlinear function, and θ t stands for the parameter to be controlled. Then, they defined the cost function, whose optimal control can be given by solving the Hamilton-Jacobi-Bellman equation. However, this study does not contain the topology of the network in the formulations. In actual neural networks, the weights of edges are trained on the basis of the comparison between the training data and estimated outputs. Furthermore, this study does not consider the variation among nodes in each layer. Motivated by this study, the author proposed the extension of ODENET to the partial differential equation (PDE) in a previous paper [3] by using graphon [4] and its extension to Sobolev spaces. Following this, we propose here a PDE-based neural network whose task is binary classification. Because of limited space, we have provided most part of the proofs of the statements in another paper.
Hereafter, we use the notation Ω = (0, 1). Let G be an arbitrary open set in a one-to three-dimensional Euclidean space. With an arbitrary T > 0, we denote G T ≡ G × (0,T), and ∂G as the boundary of G. The notation C l (G) (l ∈ N) denotes a set of functions defined on G with l-th order continuous derivatives. L 2 (G) denotes a set of square-integrable functions defined on G, equipped with the For r ∈ N, we define the Sobolev spaces H r (G), . We also introduce the anisotropic Sobolev space H 2,1 (Ω T ) ≡ L 2 (0,T; H 2 (Ω)) ∩ L 2 (Ω; H 1 (0,T)). Let BUC(G) be the space of functions that are uniformly bounded and continuous on a set G. Given T > 0, we use the notation H T ≡ (0,T)×Ω×Ω. Subsequently, we introduce two functions w 0 (x) and w 1 (t, x, y), which correspond to the weight parameters in the output and hidden layers of the neural network, respectively. They belong to L 2 (Ω) and L 2 (H T ), respectively. In these spaces, we introduce the notations of the inner product as follows: For simplicity, we also use the vector notation ì w ≡ (w 0 , w 1 ) ⊤ . The norms of vector and product spaces are defined in the usual manner. We denote the norm of the product space L ≡ L 2 (Ω) × L 2 (H T ) merely by ∥ · ∥, as long as there is no confusion.

Formulation of the neural network
Following our previous paper [3], given T > 0, we formulate the continuous version of a neural network as follows: where ν > 0, ϕ 1 (·) denotes the activation function, and ∇ 2 = d 2 dx 2 . Additionally, we impose the initial and boundary value conditions as follows: Hereafter, we consider Eqs. (1) and (2) as the continuous limit of the neural network. Furthermore, to clarify the dependence of u(t, x) on its initial value u 0 and weight parameter w 1 (t, x, y), we frequently use the notation u(t, x; N) are the labels, and X i ∈ R m with some m ∈ N, supervised learning should fit the network so that it minimizes a cost function J. In this study, we limit ourselves to the case of the binary output Y i ∈ {0, 1}, and the case of cross-entropy. Furthermore, we define the function of the output layer in our setting. Let us denote the weight parameter at the output layer by w 0 = w 0 (x), x ∈ Ω. Accordingly, the output of the fan-out layer, given the testing data X i = (X i1 , X i2 , . . . , X im ) ⊤ ∈ R m , will be the following: Accordingly, the cost function is represented as follows: To make this cost function coercive, we added the second and third terms via Tikhonov regularization. Additionally, for later convenience, we restrict the space of ì w to L (R) ≡ , with some R > 0. Next, for given T > 0, our problem is written as follows: s.t. ì w satisfies (1)-(2) with u 0 replaced by u 0(i) ′ s. (P)

Solvability of the equation
Here, we discuss the solvability of problem (1)-(2).
Theorem 2.1. Let T be an arbitrary positive number, let us assume the following.
Next, we state the existence of an optimal control ì w to Problem (P) [5].

Auxiliary lemma
Next, we prepare an auxiliary lemma.

Lemma 2.1.
In addition to the assumptions in Theorem 2.1, let us assume that ì w ∈ L (R) is given, and we introduce △ i u(t, x) ≡ u(t, x; w 1 + △w 1 , u 0(i) ) − u(t, x; w 1 , u 0(i) ). Additionally, let us assume c 0 M R < ε ′ , where ε ′ is a certain small number, and c 0 > 0 denotes the minimum positive constant that guarantees the estimate 4ν 2 t . Accordingly, △ i u(t, x) ∈ H 2,1 (Ω T ) and is represented as Proof. We briefly write the essence of the proof here. Notably, △ i u solves in Ω T , As for the form of the fundamental solution Z(t, x, y), one can refer to [7]. After applying the mean value theorem, we can obtain the explicit representation of the solution by using the Neumann series.

Necessary condition
We state the necessary condition for obtaining the optimal control to Problem (P) [6].
, and From Theorem 2.3, we have the representation of the Fréchet gradient of J at ì w.
We also assert where L 0 > 0 is the Lipschitz constant.

Numerical algorithm
Let us propose an algorithm that provides a sub-optimal solution. Mathematically, the algorithm can be obtained by regarding the weight parameter functions as elements in Hilbert spaces, and then identifying the spaces with their dual spaces [8].
Notably, the magnitude of ∥ ì w ∥ is restricted to be less than R. Therefore, we shall employ gradient projection method. We update ì w according to the following rule: where {α j } j denotes a sequence of small positive numbers, satisfying ε 1 < α j < 2 L 0 +2ε 2 ( j = 0, 1, 2, . . .) with some positive constants ε i (i = 1, 2) [9], and P R denotes a map defined on L that replaces its arguments with their projection on a ball of radius R in L: otherwise.

Conclusion
In this paper, we extended the concept of ODENet to the PDE version. We theoretically proved the existence of the optimal solution to the proposed framework. We also proposed a numerical scheme to obtain the (sub-) optimal solution based on gradient descent. However, the drawbacks of our method are some restrictions introduced to theoretically guarantee the existence of an optimal control and the explicit representation of the Fréchet derivative of the cost function. Our future study will be performed with the relaxation of these restrictions, and it will also concern the numerical implementation of the proposed method.