{'fraction_non_alphanumeric': 0.03209060877772534, 'fraction_numerical': 0.016674532011955324, 'mean_word_length': 5.024401800521203, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep neural networks for fast acquisition of aortic 3D pressure and velocity flow fields

Computational fluid dynamics (CFD) can be used to simulate vascular haemodynamics and analyse potential treatment options. CFD has shown to be beneficial in improving patient outcomes. However, the implementation of CFD for routine clinical use is yet to be realised. Barriers for CFD include high computational resources, specialist experience needed for designing simulation set-ups, and long processing times. The aim of this study was to explore the use of machine learning (ML) to replicate conventional aortic CFD with automatic and fast regression models. Data used to train/test the model consisted of 3,000 CFD simulations performed on synthetically generated 3D aortic shapes. These subjects were generated from a statistical shape model (SSM) built on real patient-specific aortas (N=67). Inference performed on 200 test shapes resulted in average errors of 6.01% +/-3.12 SD and 3.99% +/-0.93 SD for pressure and velocity, respectively. Our ML-based models performed CFD in +/-0.075 seconds (4,000x faster than the solver). This proof-of-concept study shows that results from conventional vascular CFD can be reproduced using ML at a much faster rate, in an automatic process, and with high accuracy.


Introduction
Computational fluid dynamics (CFD) has significant potential in cardiovascular settings, with applications including modelling of complex flow patterns and non-invasive estimation of vascular pressure [1]. Importantly, several studies have validated CFD measurements against conventional clinical methods such as 4D cardiovascular magnetic resonance (CMR) imaging or catheter-based measurements [2] [3]. However, the true strength of CFD lies in situations where it Figure 1: The automatic pipeline for pre-processing each patient 3D model before statistical shape modelling is performed.
Segmentation of the images is followed by remeshing, smoothing and clipping of inlets/outlets and head & neck vessels. order to maximise the "smoothness" of the deformation [17]. Therefore, deformed meshes are assumed to have nodes in 'spatially correspondent' regions of the aorta (see appendix). Principal component analysis (PCA) was used to reduce the dimensionality of the momenta, meaning that each aortic deformation field (and therefore each shape) could be expressed using only 35 parameters. This equates to an almost 15 times reduction in deformation matrix size, while maintaining 99% of the variance. The surface template computed by our SSM contained 2,541 nodes ( Figure 2). This surface template was also converted into a volume mesh with tetrahedral elements (29,000 nodes). Using the SSM, new aortic shapes were generated by randomly sampling the previously computed PCA shape mode coefficients using a Gaussian distribution. With this approach, 3000 deformation fields were created (one for every new subject). By applying these new 3D spatial deformations onto the aortic surface and volume template, 3000 surface/volume meshes were generated. Additionally, since all deformed meshes derive from the template mesh, all synthetic subjects contained the same numbers of nodes/elements. This was vital for enabling the dimensionality reduction of derived flow fields as described in later sections. The mean centreline lengths and diameters (computed using the "sphere-inscribed radius" function in VMTK) of the generated synthetic aorta population were compared to those of the real patient cohort.

Computational Fluid Dynamics Pipeline
Volume meshes previously generated with the SSM were unsuitable for CFD computation. This was primarily because low mesh skewness could not be guaranteed, and remeshing was not an option since nodes were to be preserved in order to maintain point correspondence. Therefore, separate meshes solely for CFD computation were built, starting from the surface of each aorta. Firstly, each surface was extended by 40mm at the inlet ( Figure 3). This was done to produce a flat and circular inlet upon which a velocity profile could be uniformly applied. Extending the inlet further than 40mm was avoided in order to reduce the likelihood of surface self-intersection. Following this, volume meshing with tetrahedral elements was performed [18] ( 400,000 cells on average). Element/node counts were deemed to be in satisfactory ranges after a mesh sensitivity analysis (see appendix).
CFD (Fluent, Ansys Technologies) was performed on all 3,000 synthetic cases. The same boundary conditions were applied to each simulation, as part of adopting a simple model which would reliably converge for all subjects. Laminar, steady-state flow conditions were enforced. An inlet velocity of 1.3m/s, corresponding to an average ascending aortic flow rate at peak systole, was set [19] [20]. Outlet gauge pressure was fixed at 0 Pa. Standard non-slip conditions were applied at the wall, and the fluid was assumed to be Newtonian with density and dynamic viscosity equal to 1,060 kg/m 3 and 0.004 Pa·s, respectively [21]. The set-up and simulation of all 3000 cases was fully automated.

Data Interpolation and Principal Component Analysis (PCA)
As CFD was performed on large unstructured meshes with inconsistent numbers of nodes/elements between cases, point correspondence had to be restored prior to PCA-based dimensionality reduction of the flow fields (needed for easier model training). This was done using the volume meshes previously generated with the SSM by 'shooting' on the template. Since each of these 'SSM volume meshes' inherited its properties from the template, relative nodal positions were preserved. Consequently, pressure/velocity data for all subjects was re-sampled from unstructured CFD meshes onto SSM volume meshes using a Voronoi kernel in the software Paraview ( Figure 4). This resulted in 3,000 newly resampled pressure/velocity fields, with all subjects containing 29,000 nodes in point correspondence. The data was concatenated into a feature vector and PCA was applied to reduce dimensionality. We aimed to capture 99% of variance with as few PCA modes as possible. Node concordance is restored between subjects.

Deep Neural Network Architecture
The architecture we adopted was a standard sequential, fully-connected deep neural network (DNN) with independent networks for pressure and velocity. The inputs for the model were the PCA scores of shape (reduced order deformations), which is also referred to as a 'shape vector'. The output of the trainable part of model were the pressure/velocity PCA scores (reduced order CFD field), referred to as a 'pressure/velocity vector'. A non-trainable inverse PCA layer (implemented in Keras using a lambda layer) serves to reconstruct the pressure/velocity vector into the full 3D flow field with 29,000 nodes (see Figure 5). Rectified linear units (ReLU) were used in each hidden layer. Linear activation functions were set at the output. Model implementation was done using Keras and TensorFlow 2.0.

Deep Neural Network Training
Models were built separately for predicting the static pressure and the velocity-magnitude. The loss function used for training was the mean absolute error (MAE), computed on the entire 3D flow field (i.e. after inverse PCA) rather than on the output pressure/velocity vector. This provides a more granular measure of error and effectively weights the importance of each PCA mode in the network according to the amount of variance it explains. Model optimisation was carried out using the Adam optimiser [22]. Hyperparameter tuning was conducted with a 5-fold cross-validation process to find the best model settings. The number of hidden layers, number of hidden layer neurons and initial learning rate were all explored. This was repeated for 1,000 model configurations, sampled using a tree-structured Parzen estimator (TPE) algorithm [23]. Batch size and epochs were set at 32 and 50, respectively. Hyperband pruning was used to terminate early training rounds if the model was deemed to be poorly fitting the validation data.
After completing hyperparameter tuning using 5-fold cross-validation, the model was retrained on the entire training dataset. From 3,000 subjects, 2,800 were randomly selected to make up the training set (with 10% being used for validation). The remaining subjects (200) formed the test set. Model training lasted 1000 epochs and training/validation loss was monitored. A workstation with an Nvidia GTX 1080Ti graphics card was used to perform training.

Model Evaluation
Once trained, models were evaluated on the test set of synthetic aortas (n=200). Absolute errors were computed for every node in all test cases by comparing the prediction value (ML) to the ground-truth value (CFD). Errors was then normalised according to subject CFD data range, as detailed in Liang et. al [11]. Normalisation was necessary to enable direct comparison between individual cases and also between CFD metrics, since pressure/velocity ranges widely differed per subject. Formula 1 details how normalised absolute error (NAE) is computed for pressure or velocity at a node i, belonging to a subject j. T rue and P red refer to CFD and ML values, respectively.
Mean node errors (MNAE N ) were computed by averaging NAE values across the population for each node (n=29,000). These were then plotted on the template mesh points in order to better visualise the magnitude of these errors with respect to their location. However, since NAE values are absolute errors this provides no insight regarding model bias. Therefore, a Bland-Altman plot was used to examine the bias and limits of agreement of the pressure and velocity DNNs. This was done for the overall aorta and for three separate regions; ascending aorta, transverse arch and descending aorta (anatomically defined).
Mean subject errors (MNAE S ) were computed by averaging NAE values in each subject (n=200). Cases with the best, median and worse mean subject error values were compared. A single population error for both pressure and velocity was given by averaging all MNAE S values. The relationship between shape mode scores and subject error (MNAE S ) was investigated with scatter plots and assessed using Pearson R coefficients and p values (p=0.05 considered significant).
In addition to node-based comparison between CFD and ML flow fields, an approach to compare more conventional pressure/velocity gradients was also used. In order to compute gradients, subject centrelines were used to create crosssectional planes at 99 locations along the length of each aortic 3D flow field. Average pressure and velocity values were then sampled at each plane ( Figure 6). To compare gradients, a common regression error metric was used (mean absolute percentage error) [24]. This metric is unsuitable when true values may lie very close to 0 (which is the case in the 3D velocity fields). Hence, mean absolute percentage error (MAPE) was only used for computing gradient errors. Finally, the models were tested on 10 real, patient-specific aortas with previously repaired CoA, completely unseen from the SSM and the DNN. This was done in order to validate the robustness of the models for inferring accurate flow fields on real subjects outside of our synthetic training/testing sets. CMR images of each patient were segmented.
Surfaces were approximated by non-rigid registration (applying deformations on the template) using the SSM. Deformation matrices for each case were decomposed into PCA shape vectors and passed as inputs into the DNN models.
Pressure and velocity-magnitude fields were inferred for each subject. Following this, all ten predictions were compared to their corresponding, ground-truth CFD flow fields using the MNAE S and MAPE metrics, as previously described.

Statistical Shape Modelling
PCA decomposition was performed on the deformation matrices (momenta) computed by statistical shape modelling.
Mode 1 relates to overall vessel size, mode 2 relates to ascending arch angulation/diameter and mode 3 describes how 'gothic' the aorta is (characterised by a more triangular arch). After PCA decomposition, 99% of the variance in the momenta could be represented with the first 35 modes. Some examples of the 3,000 synthetic subjects produced by randomly sampling and combining 35 PCA mode scores are shown in Figure 7. Characteristic dimensions of the synthetic aortas (length and diameter) were found to be statistically similar to those of the real patient cohort (see Table 1).

Training Data
After CFD was computed on all cases (n=3000), values were interpolated from high resolution meshes onto lowerresolution grids in point correspondence. Interpolation errors (MAPE) were computed by comparing pressure/velocity gradients in each case ( Figure 6). The mean loss in accuracy due to interpolation was found to be 0.056% ±0.027 and 0.849% ±0.247 for pressure and velocity-magnitude, respectively. The data post-interpolation was used as the 'ground-truth' training and testing sets.
PCA decomposition of the pressure and velocity training data matrices (2,800 x 29,000 each) was then performed,  Figure 8.

Model Architecture
The model input layer size was set at 35 (number of shape modes). Output layer sizes were set at 20 and 55 for pressure and velocity, respectively (number of pressure/velocity modes). Hyperparameter tuning using cross-validation was performed 1,000 times to search for the optimal learning rate, number of neurons and number of layers, with tuning taking ∼4 hours per model. Pressure and velocity model architectures as a result of the optimisation process are shown in Figure 9.

Model Predictive Performance
Pressure and velocity-magnitude fields were computed on the test set (n=200) using the trained DNNs. Inference took 0.075 seconds per subject for both DNNs. In comparison, conventional CFD took ∼5 minutes on average for convergence, demonstrating an approximate 4,000x speed-up with ML.

Node errors
Average node prediction errors (MNAE N ) were projected onto the template to visually assess the distribution of errors ( Figure 10). Pressure errors were observed to be larger in the ascending aortic region. Velocity errors were more prevalent in the underside of the arch and descending aorta. Bland-Altman analysis showed negligible prediction biases for the overall aorta ( Figure 11 and within selected regions (see appendix).  Figure 11: Bland-Altman plot for evaluating bias and limits of agreement for DNN models. All nodes (n=29,000) for each subject (n=200) were concatenated (5,800,000 data points). Each value was compared between ML and CFD values. Only 1,000 randomly selected points were graphically drawn to improve readability.

Subject errors
The population error for pressure and velocity was 6.01% ±3.12 SD and 3.99% ±0.93 SD, respectively. The test cases with the best, median and worst subject error (MNAE S ) are shown in Figure 12. The gradients of pressure/velocity for these same cases was also computed ( Figure 13). The relationship between pressure/velocity MNAE S and shape mode coefficients is presented in Figure 14. The second and third mode showed statistically significant correlations with velocity prediction MNAE S , with the third shape mode showing the highest correlation (R=-0.31). No significant correlations were found between any other shape mode and MNAE S .

Validation with real patient data
Ten patient cases with previously repaired coarctation of the aorta were collected and segmented. Surfaces were approximated through non-rigid registration (deformation of the SSM template mesh). The corresponding deformations were decomposed into shape PCA vectors. Shape PCA projections were found to be in acceptable ranges when compared to those from the original cohort ( Figure 15). Shape PCA vectors were then inputted into both pressure and velocity DNNs. Predictions were then compared against CFD simulations performed using the same pipeline as previously described (Page 2.2). The population error was found to be 10.19% ±10.41 and 4.47% ±1.18 for pressure and velocity, respectively (n=10). This corresponds to an increase in mean subject error by 4.18% for pressure and 0.48% for velocity, when compared to the test set of 200 synthetic aortas. The subjects with the median pressure and velocity MNAE S are shown in Figure 16.

Discussion
The main findings of this study were: i) statistical shape models (SSMs) and PCA are suitable for creating synthetic training data and dimensionality-reduced representations of 3D shape and flow, ii) DNNs based on these dimensionality-reduced representations can predict pressure and velocity fields with high accuracy.

Synthetic Data Generation and Dimensionality Reduction
A key element of our approach was to use an SSM and PCA to both generate a synthetic training dataset (n=3,000), and to parameterise aortic shape/flow fields for simplifying DNN model training. We demonstrated that less than 60 PCA modes are needed to capture the majority of variance within aortic shape deformations and 3D pressure/velocity fields. Mean PCA reconstruction errors of pressure/velocity were found to be low, but not insignificant in the worst observed test-set reconstructions (Figure 8). Geometric properties of our synthetic data were shown to be mostly close to the real cohort (Table 1). In the future, newer approaches for generating synthetic data and creating dimensionalityreduced representations of complex structures should be explored, notably deep-learning methods such as autoencoders [11] and generative adversarial networks (GANs) [25]. In some studies, autoencoders have been shown to be superior to PCA-based methods (e.g. for 3D facial surface reconstruction) [26] [27]. Additionally, GANs have shown promise for creating images of synthetic patients afflicted with congenital heart disease [28]. Finally, instead of sampling a Gaussian distribution to find combinations of shape PCA parameters, other methods for creating new DNN training data may be more appropriate such as Latin hypercube sampling [29].

Deep Neural Networks
Our ML models were observed to predict point clouds of both pressure and velocity flow fields with good accuracy, while being approximately 4,000x faster than our conventional CFD method. Node errors for pressure were seen to be larger at the inlet, while velocity errors were more skewed towards the distal regions of the aorta ( Figure 10). This is most likely due to the CFD boundary conditions at the inlet and outlet constraining the pressure/velocity variability at these regions. The further away from the aortic inlet or outlet, the greater the variability in velocity or pressure, respectively. It should be noted that there were no significant biases in either pressure or velocity predictions, suggesting that there were no systematic errors with the models (Figure 11). In testing, we found that the mean pressure subject error was slightly higher than the mean velocity error. This is despite the pressure PCA model capturing more variance than the velocity model. A possible explanation for the higher pressure errors is that the association between shape and pressure is more complex than that between shape and velocity in aortic domains. This is supported by the observation that shape modes do not correlate with pressure errors (Figure 14). Interestingly, there was a strong negative correlation between shape mode 3 and velocity error. This suggests that more 'gothic' aortas (characterised by a more triangular arch) were less prone to velocity prediction errors. A possible explanation may be that the gothic arch constrains downstream flow patterns (where most velocity errors occur), hence making it easier for the model to characterise flow features associated to this subset of aortic shapes.
The approach of using DNNs to model 3D aortic pressure and velocity flow fields has been described in other works [11] [10]. In this study, fully-connected DNNs were used. Other studies have used recurrent neural networks or specialised architectures such as PointNet to build CFD-based ML models [9] [30] [31]. Further DNN architectures should be tested in the future, including probabilistic architectures which enable outputting uncertainty intervals during inference [32] [30] [33]. In order to improve model performance, more training data should be generated, which will in turn enable the use of more complex models.

Limitations
In order to translate our DNN-based CFD approach to clinics, there are two main modelling limitations which need to be bypassed. The first is related to the loss in surface accuracy when using SSM representations of aortic shapes.
The second is the current simplicity of the CFD approach, which needs to be further developed in order to generate more meaningful DNN training data.

Shape Parameterisation
It has been seen in previous studies that aortic CFD flow fields are highly sensitive to geometric and topological variation [34] [35]. For this reason, using shape vectors that are accurate descriptors of the aortic surfaces is critical.
In this study, we performed ML predictions and CFD either on synthetic aortas or on SSM approximations of real aortas, both of which are completely parameterised by our shape PCA vectors. However, SSM approximations of real aortas do not match the 'true' segmentation surface one-to-one. In the future, the effect of this registration error on the resultant CFD flow fields should be investigated. Where possible, augmentations to the shape vector should be trialled in order to see if DNN prediction errors for real subjects can be reduced. This may require including additional shape information as inputs in the DNN models, such as a registration error or a geometric feature (e.g. centreline diameters). Such complimentary shape descriptors may be used to better inform the network of important features not fully captured by the shape vector alone. Additionally, multiple SSMs based upon templates other than the mean aortic shape should be generated. These would enable closer non-rigid registration for unique cases where the target aorta deviates significantly from the mean shape. Finally, more subjects should be added to our SSM in order to be able to fit a wider variety of aortic shapes.

Computational Fluid Dynamics
In this study, a simplified CFD pipeline was chosen in order to easily automate and ensure convergence for a large number of simulations (n=3,000). A standard CFD solver set-up was used, assuming steady-state conditions and incompressible flow. Previous studies have found this solver set-up to be sufficiently accurate for producing aortic pressure gradients in clinical applications [36]. Additionally, it was assumed that the flow through the great arteries during systole was laminar during peak systole [37]. In the future, Reynolds numbers may be computed for individual cases in order to properly assign turbulence models. However, for complex morphologies (such as CoA), Reynolds number has been seen to be an inconsistent measure of turbulence [38]. Some studies propose an alternative solution to turbulence modelling when performing CFD on a large scale, which involves meshing the domain with extremely high numbers of nodes [10].
Boundary conditions we selected included a fixed, flat velocity inlet and zero pressure outlet condition for all cases.
However, idealised inlet velocity profiles (flat, parabolic etc.) have been shown to be ineffective for producing clinically relevant data [39] [40]. In the future, image-derived flow profiles should be included as a DNN input parameter and as part of the synthetic CFD training data generation [41]. In this study, it was decided to omit the head and neck vessels from the CFD model for simplicity, however would be required when aiming to simulate realistic patient-specific aortic hemodynamics [35]. The inclusion of head and neck vessels along with lumped parameter outlet models such as Windkessel models should be explored in the future [42] [43], as modelling downstream resistance has been seen to produce more clinically meaningful results [44] [45] [46].

Conclusion
In this proof-of-concept study, we have proposed a pipeline for building ML-based models to perform repetitive vesselbased CFD tasks. Generation of synthetic aortic training data by means of shape modelling allowed ML techniques to be used even where data scarcity is an issue (n=67). Point correspondence was maintained between subject meshes in order to enable PCA. Our ML models were able to compute pressure and velocity flow fields much more rapidly (4,000x) than traditional CFD solvers, without large computational requirements or simulation setup. Comparison between predicted and ground truth test cases revealed good overall performance. The approach described in this study is shape-driven and is applicable to any vascular structure which can be segmented from medical images. In the future, the models should be improved so they can perform inference on prospective data from real patients. The requirements for this are two-fold; improving the accuracy of the shape representation methods, and using a more realistic CFD pipeline for generating training data. Finally, new DNN models should be tailored for specific clinical applications (such as for predicting exercise conditions).

Conflict of Interest
None.