{'fraction_non_alphanumeric': 0.031446540880503145, 'fraction_numerical': 0.012921669525443111, 'mean_word_length': 4.644078735075831, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
UCNN: A Convolutional Strategy on Unstructured Mesh

In machine learning for fluid mechanics, fully-connected neural network (FNN) only uses the local features for modelling, while the convolutional neural network (CNN) cannot be applied to data on structured/unstructured mesh. In order to overcome the limitations of FNN and CNN, the unstructured convolutional neural network (UCNN) is proposed, which aggregates and effectively exploits the features of neighbour nodes through the weight function. Adjoint vector modelling is taken as the task to study the performance of UCNN. The mapping function from flow-field features to adjoint vector is constructed through efficient parallel implementation on GPU. The modelling capability of UCNN is compared with that of FNN on validation set and in aerodynamic shape optimization at test case. The influence of mesh changing on the modelling capability of UCNN is further studied. The results indicate that UCNN is more accurate in modelling process.


Introduction
The development of artificial intelligence (AI) has brought convenience to people's life. The machine learning algorithms behind AI has also affected various industries and fields, causing a wave of technological reforms. The key factor that promotes the development of AI is the available big data, which is the natural characteristic of computational fluid dynamics (CFD). Therefore, many researchers adopt machine learning techniques to process and analyze massive amounts of CFD simulation and flight test/wind tunnel experiment data. Machine learning provides a modular and agile modeling framework that can be tailored to address many challenges in fluid mechanics, such as reduced-order modeling, experimental data processing, shape optimization, turbulence closure modeling, and control [1] . However, machine learning algorithms cannot be applied perfectly in the field of CFD. Here we are mainly concerned with convolutional neural network (CNN) [2] . As one of the most advancing and widely used methods in machine learning, CNN has achieved great success in the field of image recognition [3] , image segmentation [4] and image super-resolution [5] . Nowadays, CNN is adopted widely in image processing competitions. Khan et al. [6] reviewed the origin, innovation, development and achievements of CNN. The hierarchical feature extraction ability of CNN emulates the deep and layered learning process of the neocortex in the human brain, which dynamically learns features from the raw data [7] . Therefore, it can enlighten the understanding and knowledge of complex flow mechanisms by applying CNN to the field of CFD. In addition, higher modeling accuracy is expected to be achieved with the help of CNN.
The traditional convolutional operation is designed to process image pixel data.
Therefore, CNN can only be applied directly to the flow-field data on a uniform Cartesian mesh. At present, the application of CNN in the field of CFD is still being explored, and most researchers adopt CNN in the following three strategies: 1. The local point-to-point mapping is constructed by fully-connected neural network (FNN), which can be regarded as CNN with all filters of size equal to 1 1  . With the tensor bases constructed from local flow-field information as input features, Ling et al. [8] employed a deep FNN to predict the Reynolds stress. In addition, in order to achieve efficient aerodynamic shape optimization, Xu et al. [9] constructed the mapping from local flow-field information to local adjoint vectors by a deep FNN. In this modelling strategy, careful design of input features is required in order to implicitly exploit the features of neighbor points.
2. The flow-field information is obtained by CFD solver on a uniform Cartesian mesh. Therefore, the traditional convolutional operation can be adopted directly.
In order to reconstruct the high-resolution flow field for two-dimensional homogeneous turbulence, Fukami et al. [10] took the low-resolution velocity or vorticity on a uniform Cartesian mesh as input, and reconstructed the high-resolution data via the proposed skip-connection/multi-scale CNN. Beck et al. [11] studied the turbulence modelling via Residual CNN [12] for large eddy simulation on a uniform Cartesian mesh. Zhou et al. [13] constructed a deep CNN to learn the constitutive relation of steady-state convection-diffusion-reaction partial differential equation, and it provides inspiration for more accurate turbulence modelling. However, the requirement of using a uniform Cartesian mesh cannot be met by most industrial applications.
3. The flow-field information is projected onto a uniform Cartesian mesh for convolutional operation. In order to predict the unsteady flow field, Han et al. [14] firstly obtained the flow-field data on a uniform Cartesian mesh by projecting from data on a structured mesh. Then a convolutional long short-term memory network [15,16] is trained on the projected sample data and tested by predicting flow fields in future time steps. However, this strategy has a main problem on the scale of Cartesian mesh, which results in the inaccurate prediction of complex flow nearby the airfoil and thus the incorrect aerodynamic coefficients.
Although CNN can effectively exploit the features of neighbor nodes, the traditional convolutional operation cannot be adopted directly on the most CFD data which calculated on the structured/unstructured mesh. Therefore, the graph convolutional network (GCN), whose convolutional operation is designed for unstructured data, is introduced.
The GCN approaches can be fell into two categories, spectral-based and spatial-based. Bruna et al. [17] firstly proposed projecting graph data into spectral domain through the graph Fourier transformation, and the graph convolutional operation is defined on the spectral domain. Due to the expensive calculation cost, the later works [18,19] focus on the approximations of spectral-based GCN. However, the learned filters depend on the graph structure, which means the spectral-based GCNs cannot be directly applied to graph data with different structure [20] . In the field of spatial-based GCN, the data structures of image can be a special instance of graphs by regarding each pixel as a node. For a general graph, the spatial-based GCN produces an output of node by aggregating the features of the node and its neighbors. The developments of spatial-based GCN are introduced detailly in the survey paper by Wu el al. [21] , and spatial-based GCNs can be divided into two main categories, recurrent-based and composition-based. Different from the main categories, MoNet [22] purposed a new framework to conclude several previous works [18,[23][24][25][26] .
where  indicates the activation function, which is tanh function in this paper. K is the total number of neurons on n th layer.   n b l indicates the bias of l th neuron on n+1 th layer.
, , n l k   is the weight connecting the k th neuron on n th layer and the l th neuron on n+1 th layer. The connection weights between layers are usually stored in the matrix. As shown in Eq.(1), only the value of current i th node is used to produce the output. Therefore, FNN constructs a point-to-point mapping function on mesh.

2) UCNN
indicates the value of the j th neighbor node of i th node. Fig. 1 illustrates the forward-propagation schematic of UCNN.
where , , , , n l k i j  indicates the weight connecting the k th neuron on n th layer of the j th neighbor node of i th node and the l th neuron on n+1 th layer of i th node. Here  is the weight function with learnable parameters, which directly influences the modelling capability of UCNN. Once the learnable parameters are determined, the value of  is calculated through the relative position between i th node and its j th neighbor node.
After careful design and selection of numerical experiments, the weight function is finally determined as, where , The idea of UCNN's architecture comes from the Unet [27] as shown in Fig. 2. On the one hand, through skip connection, the low-level features and high-level features can be combined for modelling. On the other hand, it is verified that the skip connection can be beneficial for training such a deep neural network(DNN) [12] . The Normalization [28] and activation function tanh, as shown in Fig. 3.

Efficient parallel implementation on GPU
As mentioned in subsection 2.1, the neighbor weights are calculated based on the relative positions. It is time-consuming and memory-consuming to calculate the neighbor weights of all nodes repeatedly during training. Therefore, the parallel implementation is necessary. It is strongly recommended to run the program on GPU.
According to the numerical experiment, the execution speed of program on RTX2080TI is at least 50 times faster than I7-10700.
Before training, the index matrix  , angle vector  and distance vector d are obtained according to the connection relationship and coordinates of the nodes.
The pseudo code for generating these vectors and matrix is shown below, For training UCNN on a small batch of data, an efficient approach is inputting all data into UCNN to obtain the full-size output, but the loss is only calculated on the batch. The program is realized in Pytoch [29] . The adopted optimization algorithm and loss function are Adam [30] and mean square error loss (MSELoss) respectively. So far, the efficient parallel implementation for forward propagation and training has been introduced.
Next, the calculation cost of UCNN is studied in windows10 system with CPU I7-10700 and GPU RTX2080TI. The total number of mesh node for training is 13490.
The adopted architecture is shown as Fig. 2. The number of input features and output features are both 4. As shown in Table.1, the total number of learnable parameters and the consumed memory of UCNN are about twice that of FNN.

Problem description
In aerodynamic design, adjoint method [31,32] is widely used when the number of design variables is large. The calculation cost of adjoint method is irrelevant to the number of design variables, and is merely proportional to the number of objective functions. In many design optimization problems, the number of objective functions is much lower than the number of design variable. Therefore, the adjoint method and gradient-based optimization form a powerful combination [33] . The continuous and discrete adjoint equations can be derived from the continuous and discrete governing equations respectively. The discrete adjoint equation can be solved in higher accuracy.
Therefore, the discrete adjoint method [34] is adopted and briefly introduced here.
After the calculation of flow field is completed, we can obtain the objective function J according to the conservative variables U and coordinates of grid When the design variables D change, they will affect the value of the objective function J by affecting the conservative variables U and the coordinates of the grid points grid x . Therefore, the derivative of the objective function with respect to design variables can be obtained according to chain derivation rule as follows Equation (7) can be shortened to, where N D is the number of design variables. Therefore, we introduce the adjoint vector  , which can be solved by, Transposing the above formula and multiplied by Considering and linearizing the governing equation Substituting Eq.(10) and Eq.(11) into Eq.(8), we can get Therefore, once the adjoint vector  is obtained via Eq. be more than 4 10 in two-dimension case. The calculation cost of  is approximately equal to once flow computation [35] . In order to improve the adjoint-based optimization efficiency, DNN-based adjoint method is proposed as shown in Fig. 4. DNN is employed to construct a mapping function from the flow-field information   be referred in Ref [9] .

Fig. 4 Flow chart of DNN-Based adjoint method
By training on data at different conditions, DNN can accurately predict  and achieve the same optimization results as adjoint method. In this paper, we take adjoint vector modelling as the task for UCNN and FNN. The performance of these two approaches will be compared under the same architecture and training settings.

Case of invariant mesh
Firstly, the performance is studied at different Ma , different angle of attack ( ) but invariant mesh. By Latin-hypercube sampling [36] within the range of  The adopted architecture of FNN is the same as UCNN, which is shown in Fig. 2.
Both FNN and UCNN are trained in the following strategy. The training process is divided into two stages. In the first stage, the batch size, the initial learning rate and the number of training epochs is set to 13 2 , 0.03 and 800, and the second stage to 9 2 , 0.003 and 500 respectively. During the training process, when no decrease of minimum value of validation loss is seen for 40 training epochs, the learning rate is halved. In order to reduce the impact of randomness, the training is repeated 5 times. The minimum validation losses and the corresponding train losses are compared in Fig. 9.
In general, UCNN achieves a better performance than FNN on the training and validation set. When modelling by basic features (Fig. 9(a) Fig. 9(b)), the minimum validation losses of FNN and UCNN are 5 8.646 10   and 5 2.979 10   respectively. Since the spatial partial derivatives, which denote the neighbor information, are included in the PD features, the validation loss of FNN is only 2.90 times that of UCNN.
(a) Basic features (b) PD features Fig. 9 The MSE loss distribution of FNN and UCNN on train and validation data set Since modelling by PD features is more accurate, only PD features are adopted in the following aerodynamic shape optimization. The class-shape-transform (CST) methodology [37] is used to parameterize the geometric shape of airfoil. The upper surface and lower surface of the airfoil are both parameterized by 6 CST parameters.  Fig. 10(a, b) and Fig.   11(a, b) respectively. The prediction error by FNN is distributed in both near-wall area and far field, while that by UCNN is only distributed in the near-wall area with much smaller value. In Fig. 10(c) and Fig. 11(c), the gradients of drag coefficient with respect to CST parameters predicted by FNN and UCNN are compared with that by traditional adjoint method. The gradients predicted by UCNN is of higher accuracy, especially the gradient for the first CST parameter which has a major influence in the drag reduction.
where D represents design variable vector, which is CST parameter vector in this paper. The penalty function method is adopted to constrain the optimization problem as follows: 1.
where ub and lb indicate the upper and lower boundary of CST parameter.

If
, terminate the optimization; else, then turn to step 2. Fig. 12 illustrates the optimal airfoils and NACA0012 airfoil. The optimal airfoils obtained by three methods are almost at the same. The slight difference in the lower surface of optimal airfoils is insensitive to d C . However, the optimal airfoil obtained by UCNN-based adjoint method is closer to that by traditional adjoint method in the leading edge of upper surface, where the shape is most sensitive to d C . The aerodynamic coefficients of optimal airfoils are compared in Table. Table.5. The main differences when generating mesh 1 and mesh 2 are the number of nodes on the airfoil and the boundary decay. However, unlike the mesh 1 and mesh 2 generated by Delaunay method, mesh 3 is generated by advancing front method, which results in the totally different structure of mesh 3 as shown in Fig. 14.
The data obtained by mesh 1 and mesh 2 is selected as the training set while the data obtained by mesh 3 is selected as the validation set.  Since the residual value of the flow field and adjoint vector is converged to 6 10  under the same boundary conditions, the data of flow-field features and adjoint vector is obtained by three different meshes but indicates the same field. Therefore, FNN can achieve the same performance on training set and validation set, because the local mapping function constructed by FNN is irrelevant to the mesh. By comparing the performance of FNN with that of UCNN, the influence of mesh changing on UCNN can be further studied. There is a numerical error in the solution of spatial partial derivatives due to the different mesh, so only the basic features are used in modelling.
The training is repeated 5 times. Fig. 15 illustrates the MSELoss of 5 training on the three meshes. It can be predicted that the MSELoss value of FNN on the three meshes is basically the same. However, UCNN can also achieve the same modeling accuracy on all three meshes after trained on data of mesh 1 and mesh 2. The minimum validation losses by FNN and UCNN are 4 3.041 10   and 4 1.218 10   respectively.
The validation loss of FNN is 2.50 times that of UCNN. Therefore, UCNN possesses a stronger modelling capability than FNN even in the case of variant mesh. The above results indicate that UCNN has a stronger modelling capability and generalization than FNN. Therefore, UCNN is a more powerful choice for modelling task based on structured/unstructured mesh. In the following work, UCNN will be adopted in other fields such as turbulence modelling and flow reconstruction. In addition, the pooling and upsampling operators for UCNN may further improve the modelling capability, and the performance are worth exploring.