{'fraction_non_alphanumeric': 0.0374016664339245, 'fraction_numerical': 0.020714052972117487, 'mean_word_length': 4.697785439596871, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting drag on rough surfaces by transfer learning of empirical correlations

Recent developments in neural networks have shown the potential of estimating drag on irregular rough surfaces. Nevertheless, the difficulty of obtaining a large high-fidelity dataset to train neural networks is deterring their use in practical applications. In this study, we propose a transfer learning framework to model the drag on irregular rough surfaces even with a limited amount of direct numerical simulations. We show that transfer learning of empirical correlations, reported in the literature, can significantly improve the performance of neural networks for drag prediction. This is because empirical correlations include `approximate knowledge' of the drag dependency in high-fidelity physics. The `approximate knowledge' allows neural networks to learn the surface statistics known to affect drag more efficiently. The developed framework can be applied to applications where acquiring a large dataset is difficult, but empirical correlations have been reported.


Introduction
Nearly all surfaces in fluid-related industries are rough at their operating Reynolds numbers. These rough surfaces easily alter the aero-or hydrodynamic properties of a fluid system and induce unfavorable consequences; for instance, reduced stability of aircraft, increased fuel consumption of cargo ships, and reduced energy harvesting capacity of wind turbines (Gent et al. 2000;Schultz et al. 2011;Dalili et al. 2009). Therefore, it is crucial to restore or replace surfaces that have degraded past a critical level. This requires monitoring roughness on industrial surfaces and efficiently predicting their effect on the flow.
One of the most important effects of surface roughness on flow is the increased drag. This additional drag is generally expressed by the Hama roughness function (Hama 1954) or the equivalent sand-grain height. The Hama roughness function is the downward shift of the mean velocity profile induced by roughness. The equivalent sand-grain height is the roughness height from the Moody diagram (Moody 1944) that would cause the same drag as the rough surface of interest. It is worth noting that to calculate the equivalent sand-grain height, the Hama roughness function in the fully-rough regime -where the skin-friction coefficient is independent of Reynolds number -needs to be determined. To accurately determine the Hama roughness function of irregular surfaces, numerical simulations with fully resolved rough structures or experiments are needed. However, neither simulations nor experiments are cost-efficient, hence not feasible for practical purposes. To alleviate the high costs, researchers have found empirical correlations between rough surface statistics and their contribution to drag (Chan et al. 2015;Forooghi et al. 2017;Flack & Schultz 2010;Flack et al. 2020). However, no universal model has emerged that is able to accurately predict the drag from a sufficiently wide range of roughness. This is mainly attributed to the high-dimensional feature space of irregular rough structures (Chung et al. 2021).
Neural networks are known to be capable of extracting patterns in high-dimensional spaces, and therefore have been successfully used in various studies using fluid flow data (Lee & You 2019;Kim & Lee 2020;Fukami et al. 2021). Jouybari et al. (2021) developed a multilayer perceptron (MLP) type neural network to find a mapping of 17 different rough surface statistics to equivalent sand-grain height. They reported a state-of-the-art performance in predicting equivalent sandgrain heights from rough surface statistics. A total of 45 data samples, obtained from direct numerical simulations (DNS) and experiments, were used for training and validating their neural network. Having a large number of data samples to train a neural network is invariably advantageous; nonetheless, constructing a fluid database that is sufficiently large to train a network from scratch imposes significant computational or experimental cost and is oftentimes considered impractical. Therefore, for the practical usage of neural networks in fluid-related applications, a framework that allows neural networks to learn a generalized mapping from a limited number of data samples is needed.
In this study, we propose a transfer learning framework to improve the generalization ability of neural networks for predicting the drag on rough surfaces when only a limited number of high-fidelity data samples are available. Transfer learning is a method that improves learning performance by adapting knowledge learned from one domain to another (Pan & Yang 2009;Chakraborty 2021). The method has been used, for example, for self-driving cars, which can be pre-trained with data from virtual car simulators (Pan et al. 2017;Martinez et al. 2017;Akhauri et al. 2020). Obviously, the driving environment in virtual car simulators is inherently different from the real world, however, pre-training a neural network with 'approximate knowledge' can provide better initial weights for training real self-driving cars.
Similarly, for flow over rough surfaces, empirical correlations between drag and surface roughness provide an approximate knowledge of real flow physics (Colebrook & White 1937;Moody 1944;Chan et al. 2015;Forooghi et al. 2017;Flack & Schultz 2010;Flack et al. 2020). These correlations were developed by a fitting procedure of large experimental and numerical datasets. However, it is not straightforward to make direct use of these datasets, in particular because the information of surface topographies and flow statistics is not always accessible. Indeed, in many cases we have advanced our understanding of physics from the valuable information embedded in empirical correlations, such as those developed by Colebrook & White (1937) and Moody (1944).
The aim of this study is to show that transfer learning of empirical correlations can significantly improve the performance of neural networks for modeling drag on rough surfaces. Our aim is also to analyze a simple neural network with and without transfer learning to gain insight of the learning behavior, which is strongly connected to the physics of the problem. The objective of the developed neural network model is to predict drag of a particular class of roughness contained in one high-fidelty dataset, which is composed of irregular homogeneous rough surfaces. However, we foresee that transfer learning will play a central role in combining empirical relations with larger datasets from a range of sources to develop predictive models of a significantly larger class of rough walls than the ones considered here. The paper is organized as follows: the details of the developed transfer learning framework including the neural networks and datasets are explained in §2. The results of learning a mapping of surface statistics to the Hama roughness function using the transfer learning framework are shown in §3. The effects of transfer learning are analyzed and discussed in §4, followed by the concluding remarks in §5.

Transfer learning framework
The developed transfer learning framework is composed of two parts: (1) pre-training step and (2) fine-tuning step (figure 1). In the pre-training step, neural networks were trained to learn an 'approximate' knowledge of the mapping of surface statistics to the Hama roughness function. The training data is created by evaluating empirical correlation functions given the surface statistics of different surface topographies. In the fine-tuning step, neural networks were tuned to learn highfidelity physics from a small DNS dataset by adapting their pre-trained 'approximate' knowledge to the domain of real physics. In §3, we show that transfer learning of empirical correlations improves the generalization ability of neural networks when learning the drag of rough surfaces from a small amount of data. First, however, we explain the details of the neural networks and pre-training and fine-tuning steps.

Neural network
We used a variation of the MLP type neural network architecture developed by Jouybari et al. (2021). The input of the employed network is a set of 17 different rough surface statistics calculated from the surface topography of interest. The 17 input parameters are composed of 8 primary surface statistics and 9 products of the 8 primary statistics. These parameters are selected based on their perceived importance on affecting drag on rough surfaces (Jouybari et al. 2021).
Let , , and be the streamwise, wall-normal and spanwise directions, ( , ) the roughness height distribution and the total roughness plan area. Then, one may define the following measures of an irregular rough surface: In addition, we define the crest height , and finally, the fluid area at each wall-normal plane ( ). From these quantities, the 17 statistical input parameters ({ 1 , 2 , ..., 17 }) can be calculated as listed in table 1. Here, ,  ,  ,  ,  , , and indicate skewness, kurtosis, porosity, and effective slopes and average inclinations in and directions, respectively. Before training, the parameters 1 to 17 were normalized with the mean and standard deviation values calculated from the dataset of empirical correlations (see section 2.2 for the details of the empirical dataset).

Pre-training step
In the pre-training step, neural networks were trained with a large dataset obtained from empirical correlation functions between rough surface statistics and the Hama roughness function Δ + . We used three correlations; Δ + = 2.5 log + + 1.12 log + 1.47; (EMP1) Δ + = 2.5 log + 0.67 2 + 0.93 + 1.3 1.07 1 − −3.5 − 3.5; and 2.5 log 2.48 + (1 + ) 2.24 − 3.5, > 0 2.5 log 2.11 + − 3.5, ≈ 0, 2.5 log 2.73 + (2 + ) −0.45 − 3.5, < 0. As mentioned earlier, we will develop neural networks to predict drag of rough surfaces contained in a small high-fidelity dataset (see §2.3 for the details of the DNS dataset). The range of the roughness function in this dataset is 5.3 < Δ + < 8.3. The lower bound lies at the boundary between transitionally and fully rough regimes. EMP1 was derived using surfaces in the transitionally and fully rough regimes with 0.4 < Δ + < 11.4. EMP2 was modeled from surfaces with roughness function in the range of 5.4 < Δ + < 10.1. Finally, EMP3 was developed with rough surfaces in the fully rough regime, where most of the surfaces become fully rough when Δ + was over 5.5 − 6.0. Therefore, all three empirical models contain 'approximate' knowledge of the drag of rough surfaces that can help the neural networks to learn the DNS dataset in the fine-tuning step.
A large dataset was constructed by randomly generating 10,000 irregular rough surfaces and calculating the associated Hama roughness function using EMP1, EMP2 and EMP3. The surfaces were constructed using the Fourier-filtering algorithm suggested by Jacobs et al. (2017). The algorithm generates self-affine surfaces with a pre-defined power spectrum , where is the wavenumber and ℎ the Hurst exponent. The Hurst exponent ℎ = 0.8 was used as in Jacobs et al. (2017). This power-law dependence is a well-known attribute of self-affine realistic surfaces (Mandelbrot 1982;Persson et al. 2004). Figure 3a shows a few examples of the generated random surfaces. The randomness is imposed by choosing random amplitudes of the power spectrum and random phase shifts of Fourier modes. Accordingly, the surface statistics, such as the skewness and the effective slope, depend on the particular combination of the random amplitudes and phases of Fourier modes. For training and validating the neural network, 7,000 and 3,000 data samples were used, respectively. Figure 3b shows the scatter plots of the Hama roughness functions, calculated by EMP1, EMP2 and EMP3, against + , , and from the 3,000 validation data samples. The surface generation method based on the spectral density (2.2), has generated surfaces with skewness bounded between −1 to +1 and effective slopes limited to 0.35. The bounded skewness and limited effective slopes are due to the wavy surfaces generated by Fourier modes and the self-affine power spectrum defined for realistic surfaces, as in Jacobs et al. (2017). The random surfaces can thus be considered to be in the low-slope regime, where the drag tends to increase with increasing effective slope (Flack & Schultz 2014). From figure 3b, we observe different distributions, or knowledge, of the Hama roughness function from the different empirical models. The aim is to seed this diverse knowledge of empirical correlations in neural networks during the pre-training step.
Three neural networks with the same architecture and the same initial weights and biases were trained simultaneously to learn the different empirical models, EMP1 to EMP3. Let be the neural network trained with the -th empirical model, , the weight matrix connecting the -th and -th layers of , the total number of weights in , the bias vector added in the -th layer, Δ + the Hama roughness function predicted by the -th empirical model, and Δ + the Hama roughness function predicted by . The neural networks were trained to minimize a loss for pre-training: 3) The first term in the right-hand side is the mean-squared-error loss and the second is the weight regularization loss, also used in Jouybari et al. (2021). The sizes of the weight matrix , and bias vector are × and , respectively, where and are the number of neurons on the -th and -th layers.
The weights and biases of the neural networks were updated in the direction of minimizing the pre-training loss on the training data samples through the Adam optimizer (Kingma & Ba 2014). The pre-training loss on the validation data samples was also evaluated each iteration of updates. Batch sizes of 16 and 32 were used for training and validating, respectively. To provide a wide distribution of data samples in the batches, i.e., preventing neural networks from learning biased data focused on average characteristics, we imposed batches to contain 50% of the data samples with Δ + in the range 6 to 8, 25% of the data samples with Δ + in the range 4 to 6, and the remaining 25% of the data samples with Δ + in the range 8 to 10. After a sufficient amount of iterations for training and validating, the set of networks ({ 1 , 2 , 3 }) that produced the lowest pre-training loss on the validation data samples was chosen to be merged into a single pre-trained network . The learned sets of weights and biases in the three neural networks, { 1 , , 2 , , 3 , } and { 1 , 2 , 3 }, were averaged into the weights and biases of , and as , = 3

=1
, /3 and = 3 =1 /3. These pre-trained weights and biases were employed as the initial weights for the fine-tuning step (section 2.3) to transfer the knowledge of empirical correlations to the domain of high-fidelity physics.
A key factor for the performance of the fine-tuned neural network is the contents of database used in the pre-training step. If the bound of the Δ + in one's high fidelity dataset is known a priori, then by using a narrow range of Δ + for pre-training increases the transfer learning performance significantly. We provide an example of this by training the neural networks with a bound of Δ + in Appendix A. In the main text, we assume that the range of Δ + of the high-fidelity dataset is a priori unknown.

Fine-tuning step
A total of 35 DNS of channel flow over rough surfaces at ≈ 500 were conducted to construct a high-fidelity dataset. The 35 rough surfaces were generated following the method proposed by Pérez-Ràfols & Almqvist (2019). Figure 4a shows a few examples of the generated surfaces. Note that different surface generation methods were used to construct the DNS and empirical datasets (section 2.2). The method used for DNS imposes an additional constraint on the roughness height probability distribution. This constraint enables the generation of rough surfaces with distinctive characteristics. For the probability distribution, we used Weibull distributions of the form By using different shape ( ) and scale ( ) parameters, rough surfaces with non-zero skewness are generated. Gaussian distributions were used to generate rough surfaces with zero skewness. Rough surfaces generated with (2.4) imitate the roughness caused by wear (Pérez-Ràfols & Almqvist 2019). By using different surface generation methods in the pre-training and fine-tuning steps, we can verify that pre-training on one class of rough surfaces can help learn the drag on a different class of rough surfaces. Such verification is important, as in practice, artificially generated rough surfaces for pre-training would not perfectly imitate real-world rough surfaces. Details of the methodology and validation of the conducted DNS are reported in the study of Yang et al. (2021). Here, we provide a brief overview. A pseudo-spectral incompressible Navier-Stokes solver, SIMSON (Chevalier et al. 2007) with an immersed boundary method (Goldstein et al. 1993), was employed for the channel simulations. Periodic boundary conditions were used in the streamwise and spanwise directions. No-slip conditions were applied at the top and bottom planes of the channel. Roughness effects were added to both the top and bottom planes. A minimal channel approach was adopted to find a small computational domain size that can accurately reproduce the Hama roughness function calculated from a full channel simulation (Chung et al. 2015). Further details on the domain size and grid resolution are provided in Appendix B.
The scatter plots of the calculated Hama roughness function on the 35 data samples against the surface statistics involved in calculating the input of neural networks are shown in figure 4b. The generated surfaces in the fine-tuning and pre-training steps show moderately different characteristics. For instance, the surfaces generated in the fine-tuning step show higher limits of the effective slopes (up to ∼ 0.8) compared to those in the pre-training step (up to ∼ 0.35). However, note that despite the larger effective slope, the surfaces are not in the regime where drag tend to decrease with increasing effective slope (Flack & Schultz 2014). The 35 data samples were split into 6, 4, and 25 data samples for training, validation, and test, respectively. More than 70% of the data samples were used for testing, i.e., not used during the fine-tuning step, to fairly evaluate the generalization ability of the developed neural networks. Note that a total of 10 data samples were used in the fine-tuning step, and these fine-tuning data samples are completely separated from the test data samples. The data used for fine-tuning and testing the neural networks is provided in table 2. The surface statistics and the Hama roughness function in the test dataset are distributed widely in the total dataset as shown by the × markers in figure 4b.
To reduce the uncertainty arising from splitting a very small dataset into training and validation sets, we adopted an approach based on averaging many neural networks. More specifically, we employed 210 neural networks, { 1 , 2 , ..., 210 }, to learn from the 210 different data combinations derived by selecting 6 training and 4 validation data samples from the 10 finetuning data samples ( 10 6 = 10 4 = 10!/(6! 4!) = 210). The weights and biases of these 210 neural networks ({ 1 , , 2 , , ..., 210 , } and { 1 , 2 , .., 210 }) were initialized by the pre-trained weights and biases, , and . Similar to the pre-training step, the weights and biases of each neural network were updated by the Adam optimizer that minimizes a loss for fine tuning , (2.5) Here, Δ + is the ground truth, and Δ + is the roughness function predicted by . After a sufficient amount of updates, the networks with the smallest fine-tuning loss on the validation data samples were chosen to predict the roughness function Δ + on test data (section 3).

Prediction of roughness functions
We compare the predicted Hama roughness functions on the 25 test data samples that were not included in the fine-tuning step. The roughness functions are calculated by the following prediction

and
. Our focus here is on the comparison between NNTF and NN to demonstrate the effects of transfer learning. However, we also include EMP predictions in the comparison since these methods are widespread in the literature.
The performance of neural networks can vary greatly depending on the selection of data for training and validation. This is because it is generally hard to form a distribution covering the entire feature space from a limited number of data samples. By ensemble averaging the predictions from neural networks learned from different data combinations, we can partly alleviate this problem. The ensemble predictions from NNTF and NN are thus obtained by averaging the prediction Similarly, we use ensemble predictions of the empirical models {EMP1, EMP2, EMP3} in EMP as, where Δ + , is the Hama roughness function predicted by the -th empirical model. Figure 5 shows the maximum and average errors calculated from the ensemble predictions by NNTF, NN, and EMP. The predictions from NN exhibit 7.62% and 23.05% of average and maximum errors, respectively, while the corresponding error associated with the predictions from NNTF are 6.25% and 14.15%. Note that both NN and NNTF are performing better than EMP here. The most significant advantage of using transfer learning is in the reduction of the maximum error; nearly ten percent of decrease is achieved by using NNTF instead of NN. As we will discuss more quantitatively in the next section, this error reduction demonstrates the capability of NNTF in learning a better generalized mapping from a small amount of DNS data.
To further investigate transfer learning, the best and worst performing neural networks out of the 210 networks in NNTF and NN were extracted and compared in figure 6. The errors from the best performing networks in NNTF and NN (figure 6a) show a similar trend as the ensemble predictions (figure 5); both average and maximum errors are clearly reduced by using transfer learning. We also note that the ensemble predictions from NNTF (figure 5) marginally perform better compared to both the best performing neural network in NN and empirical correlation in EMP (figure 6a).
The advantage of using transfer learning is more clearly demonstrated in the worst performing case (figure 6b). The maximum error of the worst performing neural network in NN is nearly 60%, showing that this network has failed to provide a reliable prediction. On the other hand, the network in NNTF exhibits significantly smaller errors, indicating that transfer learning is helping the network to provide a safer prediction even in the worst scenario. Therefore, these results show that transfer learning enables the networks to learn a more generalized mapping from limited DNS data by adapting an 'approximate' knowledge from empirical correlations. It is also important to emphasize that ensemble predictions should be used in practice to reduce the uncertainty arising from a limited pool of data as can be seen from the wide range of errors that occur with the best and worst performing neural networks in figure 6.

Analysis of transfer learning and approximate knowledge
In the previous section, we demonstrated that transfer learning of empirical correlations improves the learning performances of neural networks. In this section, we analyze the effects of transfer learning and why 'approximate knowledge' in the empirical correlations can help neural networks.

Effects of transfer learning on weight generalization
In this section, we investigate how transfer learning improves the generalization ability by characterizing the learning weights of the networks. Let , where = 1, 2, ..., , be the vectorized elements of the weight matrices , in ( = 488 in this study). Then, we define the deviation of weights from different as /210. Note that the networks in NNTF were initialized with the pre-trained weights, and the networks in NN were initialized with the same random weights. Therefore, the deviation indicates how a weight in a neural network is updated differently for different combinations of training and validation data samples (i.e., different ). In the ideal case with an infinite amount of data samples, the deviation will approach zero as the network would converge to specific weights representing a unique generalized mapping of the real physics.
The distributions of the deviation calculated from NNTF and NN are compared in figure 7. A number of weights with large are observed in NN, indicating that the updates of weights largely deviate depending on the data. This implies that the current data is insufficient for NN to learn a generalized mapping. On the other hand, the deviations are significantly reduced in NNTF, implying that the networks in NNTF are learning a better-generalized mapping compared to NN. This is because the networks in NNTF were initialized with the pre-trained weights containing the 'approximate' knowledge. These weights provide a better initial state for the weight optimization problem in the fine-tuning step.

Effects of transfer learning on sensitivities of input surface statistics
This section investigates how transfer learning affects the neural networks in learning the sensitivities of the input surface statistics. The sensitivities of input surface statistics in predicting  drag are obtained by calculating the derivatives of drag with respect to each input (surface statistical measure). A similar analysis was employed by Kim & Lee (2020), for evaluating the influences of flow variables in predicting heat transfer characteristics. We define the sensitivity of the input in predicting drag Δ + (defined in equation 3.1) as: Here, · is the averaging operation along with the calculated derivatives from the 25 test DNS dataset. The derivative Δ + can be analytically calculated as neural networks are composed of differentiable function compositions and matrix multiplications. However, more efficiently, it can be done by using an automatic differentiation algorithm. In this study, we used the automatic differentiation algorithm implemented in PyTorch (Paszke et al. 2017). Figure 8 shows the sensitivities of the input surface statistics when trained with NNTF and NN. It is observed that for both neural networks, 12 = × , 10 = × and 6 = have the highest sensitivities. In other words, these are the most important surface measures when it comes to predicting the friction drag for the particular class of rough surfaces contained in the high-fidelity dataset. These statistics are not explicitly included in the empirical models (EMP1), (EMP2) and (EMP3). Therefore, this indicates that the neural networks are learning the mapping of surface statistics to drag beyond the empirical models. However, the importance of 12 , 10 and 16 is partially implied by previous studies. The importance of the product of effective slope and skewness is partly implied in the study of Forooghi et al. (2017), as their empirical model (EMP2) is nonlinear with respect to skewness and effective slope. The importance of porosity in predicting drag was recently emphasized by Jouybari et al. (2021). Although the result shown here is for a particular class of rough surfaces, it demonstrates the necessity of expanding the feature space of surface statistics for drag prediction to more complex nonlinear combinations of the most basic statistical moments.
To understand what influence transfer learning has on sensitivities, we can identify from figure 8 the input statistics that are ranked differently by NN and NNTF in terms of importance. The surface statistics related to effective slopes and skewness, e.g., 12 = × , 10 = × , 2 = , 4 = , and 5 = , are found to be more sensitive in NNTF. These higher sensitivities in NNTF are mainly due to the inclusion of the approximate knowledge about the drag dependencies on effective slope and skewness in the empirical correlations. Conversely, the sensitivity of the input defined as the square of skewness ( 17 = × ) is smaller in NNTF compared to NN. This indicates that NNTF learns that the square of skewness compromises the information about the sign of skewness, which is known to be important for drag prediction (Jelly & Busse 2018). In addition, NNTF reduces the sensitivities of statistics related to kurtosis ( 3 = , 13 = × , and 11 = × ). As kurtosis measures the outliers (DeCarlo 1997), the reduction indicates that the effects of the outlier roughness heights are relaxed in NNTF.

Approximate knowledge in the empirical correlations
As briefly introduced in § 2.2, the empirical correlations contain different types of approximate knowledge. This is because the empirical correlations were fitted from different types of surfaces (Chung et al. 2021); EMP1 was derived from data of a pipe with sinusoidal irregular roughness; EMP2 was developed from data of rough surfaces with different arrangements and size distributions of roughness elements and finally; EMP3 was constructed from a wide range of realistic rough surfaces, such as gravels and commercial steel pipes. As a consequence, each of the relations EMP1, EMP2 and EMP3 depends on a different measure of roughness height ( + , + , or + ) and different combinations of surface statistics ( and/or ). Figure 9 shows contour levels of the drag predicted by (EMP1), (EMP2) and (EMP3) in a map spanned by skewness and effective slope . Note that the roughness heights for EMP1 to EMP3 ( + , + , and + ) are chosen by their respective values that produce Δ + = −3.5 at a common location ( = 0.3 and = 0) as in Chung et al. (2021). The correlation EMP1 contains the approximate knowledge that drag tends to increase with increasing effective slope, while it does not contain any dependency of drag with respect to skewness. This is because EMP1 was derived from fitting data with a narrow regime of skewness as shown in the black • markers in EMP1 of figure 9. Oppositely, the correlation EMP3 contains the knowledge that drag tends to increase with increasing skewness, without the dependency on effective slope. This is because the majority of the fitting data for EMP3 (black • markers in EMP3 of figure 9) lies on an insensitive regime of effective slope -between the sparse regime ( < 0.3−0.6) and dense regime ( > 0.4−3.0) -with relatively small effects on drag (Jiménez 2004). EMP2 shows a nonlinear dependence of drag on the skewness and the effective slope. As a result, this model includes approximate knowledge about the drag dependency on the product of skewness and effective slope as also discussed in §4.2.
As the empirical correlations are derived by fitting data from a particular set of surfaces, a naive extrapolation of them for predicting drag of other set of rough surfaces may lead to large errors. For example, if the correlations are directly applied to predict drag on the current test surfaces of the DNS dataset (red × markers in figure 9), the average and maximum errors of EMP1, EMP2, and EMP3 are (17%, 29%), (8%, 17%) and (25%, 44%), respectively. Note that F 9. Contours of the roughness function calculated from EMP1, EMP2, EMP3. The roughness heights ( + , + , + ) = (0.234, 0.474, 1.106)) are used for calculating the empirical correlations. The red × and • markers indicate the test and fine-tuning DNS data samples, respectively. The black • markers indicate the fitting data samples used for deriving the empirical datasets (data extracted from Chung et al. (2021)).
the errors from EMP2 are the smallest due to the similar surface spaces between the current DNS data and the fitting data. Interestingly, large errors of the empirical models do not necessarily lead to inaccurate fine-tuned neural networks. To demonstrate this, we fine-tuned neural networks pre-trained only with one empirical correlation. We found that the average and maximum errors of the ensemble networks fine-tuned from EMP1, EMP2, and EMP3, were (6%, 18%), (6%, 17%), and (7%, 14%), respectively. Despite the high errors of the empirical correlations of EMP1 and EMP3, their resulting fine-tuned networks are as effective as the fine-tuned network from EMP2. This indicates that an empirical model for pre-training does not necessarily need to be accurate, it merely needs to provide an approximate knowledge of the dependencies of on surface statistics. Moreover, NNTF, which is pre-trained with all of the approximate knowledge provided by the three empirical models, achieves a more favorable overall performance (figure 5), compared to the fine-tuned networks pre-trained with any single EMP. This is because the knowledge contained in each empirical model contributes with information of drag dependencies. This can be qualitatively shown by visualizing knowledge domains. Figure 10 compares the knowledge domains of high-fidelity physics (DNS domain), approximate knowledge in the empirical models (EMP domain), and non-physical knowledge from the randomly initialized neural networks (Non-physics domain). To visualize the knowledge domains, we extracted principal axes of the data samples composed of ( , , Δ + ) in each domain. Note that Δ + in the DNS, EMP, and Non-physics domains are obtained from simulations, empirical models, and randomly initialized neural networks, respectively. The DNS domain is composed of rough surfaces in the DNS dataset, while the EMP and Non-physics domains are composed of rough surfaces in the empirical dataset. After computing the principal axes in each domain, ellipsoids that approximately bound the DNS, EMP, and Non-physics domains along their principal axes are visualized. As shown in figure 10a, no particular correlation (or rather alignment) can be found between the Non-physics domain and the DNS domain. On the other hand, an alignment between the EMP domain and the DNS domain is observed in figure 10b. Therefore, it is expected that neural networks with approximate knowledge from the EMP domain can more easily adapt to the DNS domain compared to those without any physical knowledge.
We also quantified the alignment between the knowledge domains by calculating the angles between the principal axes. The angles between the principal axes in the EMP and DNS domains are computed as (16.5 • , 18.1 • , 8.9 • ), while the angles between the principal axes in the Nonphysics and DNS domains are computed as (120.9 • , 119.9 • , 12.3 • ). Accordingly, the aligned 'approximate knowledge' from EMPs assists the networks to adapt to the DNS domain; thus,

Conclusions
We have developed a transfer learning framework to learn the Hama roughness function from a limited number of DNS data samples. The framework is composed of two steps: (1) pre-training step and (2) fine-tuning step. In the pre-training step, neural networks learn an 'approximate' knowledge from empirical correlations. In the fine-tuning step, the networks are fine-tuned using a small DNS dataset. Neural networks trained with transfer learning have shown a significant improvement in predicting the roughness function on test data samples not included in the fine-tuning step. This is because the 'approximate' knowledge in empirical correlations is aligned to high fidelity physics, which assists neural networks to learn better-generalized weights. In addition, a sensitivity analysis showed that the neural networks with transfer-learning were clearly emphasizing the importance of certain input surface statistics (effective slopes, skewness and porosity) in predicting drag. These extracted statistics are in general good agreement with what has been reported by other investigations, but also highlight that nonlinear functions of several surfaces statistics could provide high correlation with drag.
We have shown that the prediction performance is enhanced when the 'approximate' knowledge of the empirical dataset is well aligned to the high-fidelity dataset. Therefore, it is advantageous to employ empirical correlations developed for classes of rough surfaces that are similar to one's high-fidelity dataset. The current NNTF, trained with irregular homogeneous rough surfaces, would not be effective when predicting drag on regular rough surfaces (e.g., cuboids, bars, etc.) or inhomogeneous rough surfaces. Similarly, predicting values of roughness function beyond training is also expected to not be effective due to the general limitations of data-driven methods in extrapolating.
To further increase the generalization ability of neural networks, ultimately, the dataset has to be expanded. A large database would also help to study classes of rough surfaces with undiscovered empirical correlations, where pre-training is currently not possible. Accordingly, a collective community effort is needed to construct and train Big Data of flows over many classes of rough surfaces in different flow conditions. One step in this direction, is the online database Roughness database (Engineering and Physical Sciences Research Council & University of Southampton 2020) currently under construction. Moreover, new primary surface statistics that strongly affect drag may be discovered in the future by extending the sensitivity analysis in §4.2 with a larger surface parameter space and Big Data. Also, expansion of the network structure would enable one to fully leverage information inside large databases. The structure of the neural network in this study is very simple, as our focus has been to show that transfer learning can improve the performance of a given network. Thus, this network structure should not be considered as the optimal model for predicting drag. In addition to expanding the database, further optimizing the neural networks will also improve the performance for drag prediction.
Finally, while we have considered transport of momentum (i.e., drag coefficient), empirical relations are widespread also for characterizing transport of energy (Nusselt number) and mass (Sherwood number). Therefore, the proposed framework can also be applied to other engineering applications where only a limited amount of high-fidelity data is available, but a significant amount of knowledge has been accumulated.
Here, we provide an example of training the neural networks with bounded Δ + . If the bound of Δ + in one's high-fidelity dataset is known, the transfer learning performance can be improved by accordingly bounding the Δ + for pre-training. In this example, we used the bound of Δ + > 6, as the surfaces in the current DNS dataset asymptotically reach the fully rough regime when Δ + ≈ 6 (Yang et al. 2021). We excluded the data samples with Δ + < 6 from the 35 DNS data samples (figure 4). As a result, a total of 29 DNS data samples are used in this section. The 29 data samples were split into 5, 3, and 21 data samples for training, validation, and testing. Note that more than 70% of the data samples were used for testing as in §3. The Δ + of the empirical dataset is bounded to Δ + > 6 as the DNS dataset. After pre-training neural networks with the bounded Δ + , we fine-tuned a total of 56 neural networks, which learn from the 56 different combinations of 5 training and 3 validation data samples out of the total 8 fine-tuning data samples ( 8 5 = 8 3 = 8!/(5! 3!) = 56). The average and maximum errors on the 21 test data samples predicted by the ensembles of (1) NNTF with bounded Δ + (NNTF-BU), (2) NN with bounded Δ + (NN-BU), and (3) EMP with bounded Δ + (EMP-BU) were (5.49% and 14.60%), (19.61% and 45.61%), and (13.22% and 27.07%), respectively (figure 11). The errors from NN-BU were notably increased compared to the errors from NN (7.62% and 23.05%, figure 5). This is because the number of data samples used for training NN-BU has decreased by ∼ 20% compared to the number used for training NN. However, despite the decrease of the number of fine-tuning data samples, NNTF-BU shows similar errors compared to NNTF (6.25% and 14.15%, figure 5). Accordingly, NNTF-BU achieved nearly 15% and 30% decrease in error percentages compared to NN-BU, which is more significant than those achieved from NNTF (figure 5). Therefore, the transfer learning performance can be improved when the range of Δ + of surfaces in one's high-fidelity dataset is a priori known. It is also worth mentioning that the errors from EMP-BU are similar to those from EMP (12.30% and 27.07%, figure 5), which indicates that the employed empirical models are reasonably valid even for Δ + slightly under 6.

Appendix B. Grid resolution and domain size
To determine the appropriate mesh resolution in our simulations, a grid independence test was conducted. We conducted the test for the rough surface with the smallest Taylor microscale ( ) along with the 35 DNS data samples. The definition of the Taylor microscale proposed by Yuan & Piomelli (2014) is adopted in the current study. We studied the grid independence using five different resolutions in a minimal channel domain (2.0 × 2.0 × 0.4 ): Case 1, (192×401×36); Case 2, (256×361×48); Case 3, (256×401×48); Case 4, (256×451×48); Case 5, (300×401×60). The calculated mean velocity ( + ) profiles are shown in figure 12. The profiles from the five different cases are found to be nearly identical in both the inner and outer layers. Thus, in the current study, we used the grid resolution of Case 3. This resolution corresponds to Δ + < 5 in both streamwise and spanwise directions. In addition, with this resolution, the grid sizes were at least four times smaller than the Taylor microscales of all surfaces in both streamwise and spanwise directions ( /Δ > 4.5), satisfying the grid resolution constraint introduced in Jouybari et al. (2021).
The sensitivity of the domain size was also studied. We performed additional 12 full channel DNS to inspect the feasibility of using minimal channel DNS in the current study. For a given channel half height , the domain and grid sizes for the full channel simulations were, respectively, (8 × 2 × 4 ) and (900 × 401 × 480), in the streamwise, wall-normal, and spanwise directions.
The smallest domain and grid sizes for the minimal channel simulations were (2.0 ×2.0 ×0.4 ) and (256 × 401 × 48), respectively. The surface statistics of the rough surfaces were reasonably well converged in both domains. For instance, the maximum differences of + , , and between the full and minimal channel domains for the rough surfaces were 0.038, 0.025, and 0.006, respectively. The resulting average and maximum errors between Δ + calculated from the full and minimal channel domains were 1.5% and 4.6%, respectively. As these errors are notably smaller than those from the predictions of neural networks ( §3 and Appendix A), we found the current minimal channel approach sufficient for our purposes. The full details of the surfaces and simulations are available in Yang et al. (2021).