{'fraction_non_alphanumeric': 0.025638754815569233, 'fraction_numerical': 0.007815613514590621, 'mean_word_length': 4.972890769637662, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

The choice of convolutional routines (primitives) to implement neural networks has a tremendous impact on their inference performance (execution speed) on a given hardware platform. To optimise a neural network by primitive selection, the optimal primitive is identified for each layer of the network. This process requires a lengthy profiling stage, iterating over all the available primitives for each layer configuration, to measure their execution time on the target platform. Because each primitive exploits the hardware in different ways, new profiling is needed to obtain the best performance when moving to another platform. In this work, we propose to replace this prohibitively expensive profiling stage with a machine learning based approach of performance modeling. Our approach speeds up the optimisation time drastically. After training, our performance model can estimate the performance of convolutional primitives in any layer configuration. The time to optimise the execution of large neural networks via primitive selection is reduced from hours to just seconds. Our performance model is easily transferable to other target platforms. We demonstrate this by training a performance model on an Intel platform and performing transfer learning to AMD and ARM processor devices with minimal profiled samples.


INTRODUCTION
Deep learning has triggered a machine learning renascence. This is now powering numerous applications, such as image classification, image segmentation, voice recognition, machine translation and others, which benefit billions of people. However, these techniques are mostly designed for cloud computing on the server side. This strategy is now increasingly scrutinized due to growing data privacy concerns and needing to operate on devices with sporadic Internet connectivity.
We are seeing a sustained adoption of deep learning in edge computing. This is encouraged by increasingly more powerful hardware that can support complex applications locally, such as phones and smart home devices. However, injurious use of powerful hardware is still limited in the breadth of workloads it can support and the rate of energy budget depletion on batter powered devices. Running deep neural networks at the edge on these devices requires compute adaptation to boost their efficiency.
Designing computationally efficient networks is one active research area in machine learning [15]. Already established large networks are often considered for deployment on smaller devices, by using quantization and model pruning [21] adaptations. But these methods reduce the original classification accuracy of those networks. An alternative approach to improve the execution of a neural network is to perform primitive selection at each convolutional layer in the network. This retains its original inference accuracy. These primitives represent alternative implementations of the convolutional operation, each producing an equivalent data processing result, but different in performance (execution speed).
The primitive selection process starts with a large set of available implementations -the primitives. Convolutional primitives include im2col with matrix multiplication, direct convolution, winograd techniques and others. These are characterized by the algorithm they use, data input and output organisations, system characteristics (vectorization, etc.). The goal is to determine the best primitive and its data transformation for each layer configuration of the network, which produces the fastest execution on a given device.
However, the primitive selection method requires a lengthy profiling stage to measure the actual execution time of each primitive in the network configurations on the target platform. This makes primitive selection difficult to apply in most applications as network layer configurations are quite varied. Consider developing at scale a machine learning-powered application on a smartphone. To optimise the network configurations, profiling is required on all possible CPUs and GPUs available on the market. This is unimaginable for a one-off calibration in the factory, so profiling is done when installing the application instead. Each additional application that requires the use of their own efficient neural network would go through the same profiling stage. In our experience, the profiling of all the primitives with the layers of very large neural networks (ResNet), can take up to hours.
Here, we ease the highly expensive profiling stage (often measured in hours [1]), by replacing it with a performance model. Two multi-layer fully-connected neural network architectures are evaluated as our performance model, which we show outperform simpler regression models. These are also ideal for transfer learning. We show that once a performance model is trained on a given platform, very few additional sample points are required from the target platform (different architecture than original platform) to transfer the performance model.
Intuitively, this eases the task of hardware vendors by building a performance model at the factory only once for each platform. This performance model can be easily ported to other subsequent devices via transfer learning. When an application registers its neural network to run on the device, our performance model will be used to find the best configuration of primitives for the network. The optimisation of a convolutional neural network with our performance model takes just milliseconds, contrasting with to hour-long exhaustive profiling of primitives on the device for each new network. This paper makes the following contributions: (1) We design a performance estimation model that accurately predicts the execution time of convolutional primitives for any configuration of a convolutional layer. (2) We show that using a performance model for primitive selection offers a significant speedup in neural network optimisation compared to profiling the device. (3) The performance model are shown to be easily transferable across platforms (hardware architecture), with efficient estimations on AMD and ARM architectures after pre-training the model on an x86 architecture.
The rest of this paper is organized as follows. Section 2 describes the primitive selection process and the previous work on performance modeling of neural networks using machine learning. The methods for performance modeling and primitive selection used in this work are discussed in section 3. The experiments are described in section 4 and their results are discussed in section 5. Finally, section 6 concludes this work and provides some future work directions.

BACKGROUND AND RELATED WORK
Multiple methods exist for reducing the computational requirements of neural networks. Solutions include reducing the floating point precision during inference [5], neural architecture search [8,12,18,25] and specialized neural network compilers aiming to generate highly efficient machine code [26]. Primitive selection is a particularity attractive solution for its simplicity. Any existing network can be optimized using primitive selection without any application knowledge.

Primitive Selection
Primitive selection for a neural network means selecting which implementation -or primitive -to use for each of the layers in the network. The assignment is made at layer level such that the entire running time of the network is optimized. The running time of a primitive depends on the specific layer configuration and the hardware on which it is running. This is further complicated by the fact that the various primitives can differ in the input and output data organisation. This captures the additional cost of 'data-layout transformations' between layers. Figure 1 represents schematically the process of primitive selection on a three-layer neural network. This chooses between primitive implementations , and each with costs , and respectively, and data transformation between layers when changing the primitive. Previous work on primitive selection focuses on optimizing the execution time of convolutional neural networks by optimizing the primitives of each convolutional layer [1,6]. Focusing only on the convolutional layers is justified by knowing that these layers incur more than 90% of the execution time of modern neural networks [27]. The problem is roughly solved in two stages. First, all primitives are profiled on the target platform for all layers of the network (the profiling stage). Depending on the network size and the target platform, this process can be very time consuming. Afterward, the execution time information is used to optimize the primitive selection (the optimization stage) for the entire network. Figure 1: Primitive selection illustrated for a three-layer convolutional neural network where each layer can be implemented with three primitives (A, B and C). This gives 27 possible implementations for the overall network. The nodes and edges have associated costs ∈ R 3 (the running time of the primitives) and ∈ R 3×3 (the running time of the data layout transformations) respectively. Primitive selection aims to assign a primitive to each layer such to minimize the total sum of node and edge costs. In the figure, primitive B is selected for the first two layers and primitive C for the final layer. This gives a total cost of The solver approach. In previous work [1], the optimization stage is performed by an off the shelf Partitioned Boolean Quadratic Programming (PBQP) solver [9]. This models the optimisation task as a graph traversal problem, in which primitives are represented by nodes with a cost and data transformations as edges with cost. For moderately sized networks such as Alexnet [19], VGG-Net [22] and Googlenet [23], using the profiled data, a PBQP-solver can give optimal results in less than a second. Final solutions are shown to provide speed-ups of up to multiple times over common frameworks. Memory footprint has also been considered in the optimisation process with the Integer Linear Programming solver [28,29].
The reinforcement learning approach. This optimisation problem has also been explored by using reinforcement learning [6]. This greatly reduces the search space by introducing heuristics about the cost of similar primitives or scaling cost on the size of the layer. The limitation of this approach is that some primitives may behave very similar on one platform, but may have completely uncorrelated behaviour on other hardware or between layer sizes, which may lead to suboptimal solutions. By comparison with a stock optimisation library, the reinforcement learning approach showed a speed improvements of 1.4 ×. However, there is no guarantee that the same heuristics would work on a different platform, so the additional cost of tuning the human-driven heuristics is still prohibitive.
Both works [1,6] rely on a lengthy profiling stage, which can make the method impractical to scale. This work aims to remove the need for a profiling stage by introducing a performance model to predict the execution time of each primitive instead on a given hardware.

PBQP
(iii) (iv) Figure 2: The primitive selection process when using a neural network based performance model. denotes the number of layers of the CNN. denotes the total number of primitives and data-layout transformations (DLT). The four steps are as follows. (i) the configurations of the layers (input number of channels , input size , kernels number of channels , kernel size and stride ) are extracted. (ii) the configurations are given to the performance model which outputs, for every configuration, the predicted primitive and DLT running times . (iii) the running times are given to a PBQP-solver which outputs the optimal primitive assignment for each layer for the network. (iv) the primitive assignment is used to optimize the running time of the original CNN. Note that the performance model is batched -performing the computation for all layer configurations simultaneously -making its inputs and outputs vectors of size .

Machine Learning for Running Time Prediction
Previous work has shown that machine learning methods can predict the running time of algorithms via performance models [14,20]. Neural networks are especially suited for this task [20]. Some effort has been invested in predicting the execution time of convolutional networks also [4,17]. Both linear regressions with polynomial features [4] and fully connected neural network [17] have been shown to accurately predict the running time of a convolutional network. These are trained and evaluated for a single default primitive and never considered for different primitive candidates. Transfer learning methods have also been shown to significantly reduce the required number of data points when creating a performance model for benchmark workloads to match the characteristics of a new platform [20]. This shows that transfer learning works well for benchmark performance models because of the wide variation between benchmarks, but the task we explore here is harder because of the subtle performance difference across various primitives.
No other work has approached deep neural network optimisation at the primitive level via performance modeling, which is the scope of our work here. In the following section we present our approach.

CNN PERFORMANCE OPTIMISATION
As mentioned in Section 2, the main drawback of primitive selection methods is the requirement of a lengthy profiling stage to gather the execution cost of each primitive and data transformation. This work aims to replace the need for a profiling stage before the primitive selection with a performance model that can predict the execution time of primitives and data transformations. We adopt a neural network based performance model due to its characteristics to capture subtle differences in performance. We perform this estimation with a model trained for the target device and pass the estimated costs to a solver for primitive selection for the convolutional neural network.
Given information about a convolutional layer (i.e. the input size, the stride, padding, kernel size, number of kernels) the performance model predicts the execution time for all primitives for this layer. Repeating the process for all layers of the network, we generate the graph required by the PBQP-solver [9] to optimise the primitive selection across the network for best performance.
The entire primitive selection process is shown in Figure 2: i Given a convolutional layer, the parameters of its configuration are extracted. ii The layer configuration information is used by the performance model to estimate the execution time of each primitive and each data-layout transformation. iii The execution time predictions are passed to the PBQP-solver to produce the primitive selection across the network. iv The primitive assignment is used to produce the final implementation for the optimised convolutional neural network.

Convolutional Layer Optimisation
There are various ways in which a two-dimensional convolution can be performed. We consider the layer parameters: width , height ℎ and number of channels of the input; kernel size , stride and number of kernels . For simplicity, we assumed that the input is square, which is common in most neural networks. The input size is notated as im (im = = ℎ). A significant number of highly efficient implementations of the convolutional operation have been proposed over the year, each with their drawbacks and benefits. Some primitives may be restricted to just some data shapes, as × ℎ × whereas others may assume input images in a different format, ℎ × × . The output shape also varies between primitives, which is not necessarily the same as the input shape. The major family of implementations (each which potentially multiple implementations) considered in this work are direct-sum2d, im2, kn2, winograd, conv-1x1 and mec. A list of all the primitives we used is presented in the appendix (table 6).
The direct-sum2d family computes the convolution through six nested for-loops, three to cover all output values ( , ℎ and ) and three for all input values that influence the specific output value ( , ℎ and ). Given the lack of optimized routines, the direct-sum2d is often one of the slowest primitives with general compilation. The im2 (or image to) family, on the other hand, performs a convolution by reshaping the input data and kernels such that the convolution is performed through a single matrix multiplication. This can greatly speed up the running time of the convolution [16]. As there is substantial data replication, this can be quite memory intensive, but we are not concerned about the memory space here, although data access is captured in the primitive execution cost. The kn2 (or kernel to) primitives aim to reduce this memory inefficiency by not performing the entire convolution as a single matrix multiplication. Instead, the entire convolution is broken down into the sum of multiple matrix multiplications [2]. The downside for the kn2 family is that it is not efficient for larger strides convolutions.
The winograd primitives perform the unstrided convolution with the least amount of multiplications possible [3]. Although this can offer significant speed-ups, it is difficult to anticipate when this is the case without profiling. We have winograd primitives specialized for different kernel sizes: 3 × 3 and 5 × 5. For kernels of size 1×1, the entire convolution is essentially a single element-wise matrix multiplication. The conv-1x1 family implements them as such. Finally, the mec (or memory-efficient convolution) primitives are two primitives that optimise memory requirements of a convolutional layer [2]. This is most often in detriment of execution time, but also occasionally on-pair with the other performance driven primitives.

Convolutional Primitive Execution Time.
To train a machine learning model to predict the execution time of primitives, we need a dataset of some data points representing the execution of primitives on a target device. We collect these data points by profiling many convolutional layer configurations. Let the execution time measured for primitive be , then our dataset is constructed as follows: where left side is the layer configuration ( , , , , ) and right side the measured times for primitives profiled in hardware. Not all primitives work for every configuration (e.g. a primitive may require a specific kernel size), hence some can be undefined.
We identify the common ranges of parameters found in a wide pool of networks, as presented in Table 1. We consider input images of up to 299 by 299 pixels -the largest input size used for ImageNet [7]. Over all these possible configurations, the total space size is roughly 20 billion different variations. Some configurations may not occur in modern convolutional networks. For example, the image size tends to reduced further in the network where the number of convolutions are higher. But we avoid to constrain the search space abruptly with arbitrary heuristics, leaving the search space largely available for a more general solution. The specific data points are selected as follows. First, a set of , and triplets are collected as they occur in a large variety of common architectures (table 7). This results in 475 unique triplets. Then, each of these triplets is combined with all combinations for the tuples and (as listed in  For our list of primitives, we adopt the primitives highlighted in other primitive selection works [1]. This includes a wide range of primitives across all the major algorithm families discussed in Section 3.1. We ignored 'fast Fourier transform' primitives as our initial experiments showed them to be the slowest for almost all layer configurations. A full list of considered primitives can be found in the appendix (table 6).

Data Layout Transformation Cost.
Convolutional primitives have different requirements for the shape of input data. If one primitive outputs the data in a different format than what the following layer requires, this data needs to be transformed to the suitable format. This adds an additional cost to using those two consecutive primitives. As such, a solver that optimizes the primitive selection needs the execution times for data-layout transformations to determine the optimal selection across the network. This set of transformation costs can be profiled on the hardware for any shape and size of data passed between layers. This cost depends only on the data size ( and ) and data-layout, with fewer candidates than for the primitive profiling. These profiler measurements are represented in our dataset as entries of the form: where , is the execution time to perform a transformation from format , with ∈ [1, ], to format , with ∈ [1, ], and is the total number of data-layout transformations. For our set of primitives, there are three different data layouts: × × , × × and × × . This results in nine data layout transformations to profile, which includes the identical transformation (to self) with cost zero. All these measurements are profiled on a target hardware to form our data layout transformations dataset.

Performance Modeling
Performance Metrics. We assess the quality of performance estimation using the median relative absolute error (MdRAE). This gives a measure too which extend a predicted performance time is expected to be off. The relative absolute error is defined as follows: |^− | where^represents the prediction and the actual measured value.
Performance Model Architectures. Given the non-linearity of most primitive families, a multi-layer fully-connected neural network is our choice for a good performance model. We use two approaches to construct the neural network architecture (Figure 3). The first (NN1), using a separate model for each primitive and for each data transformation. In Figure 3, these are indicated as 1 , where is the order of primitives. In the second (NN2), we use a single model to estimate the performance of all primitives. Both of these models take as input the parameters defining the shape of a convolutinal layer, as discussed in previous sections.
Step ii) of Figure 2 schematically represents our second approach, estimating all costs at once. We emphasise NN2 because having a single model to estimate all performance costs makes deployment and logistics much simpler. We also have to consider data-layout transformations, so a similar network is trained to predict the costs of these data transformations. Using NN2 also reduces the training time as many similarities exist between primitives, which are automatically determined during training, thus reducing the amount of data required for training.
Loss Function. Predicting performance is a regression task. As such, we use the mean squared error (MSE) loss as the loss function for network output. However, some execution times may be undefined, for instance when a primitive cannot be applied to a particular shape of a convolutional layer. When training NN1 for each primitive, we filter out the undefined conditions. When training with NN2, for estimations of performance across all primitives, training sample points can have both actual values and undefined labels. We make sure these have no effect on the training by masking the values and gradients of those primitives with undefined values in the forward pass and in the back-propagation stage respectively. This causes their squared error to equate to zero, resulting in no influence on the training process.
Data Point Normalization. The performance times can be substantially wide in magnitude. With our loss function, MSE, favorable estimations are observed in the larger values. To address this issue, we adopt the log expression of performance values when training the network, to scale favorably both large and small values. Finally, to improve the training of the neural network, all input and output data were standardized (i.e. transformed to have zero mean and a standard deviation of one). Performing all these steps would result in the following transformation for a given variable : where˜denotes the normalized result and¯the mean of .

EXPERIMENTS
This section introduces the experiment setup for performance data collection. This is followed by preliminary observations on this data and how a performance model is calibrated. We also design experiments to validate that primitive selection can operate on predicted performance values and that a trained model can be transferred to use in other hardware.

Primitive Performance Profiling
4.1.1 Experimental Environment. We run our data collection on three machines profiling the execution of all primitives. An Intel Core i9-9900K @ 5.0GHz, an AMD A10-7850K @ 3.7GHz and an ARM Cortex-A73 (rev2) @ 2.36GHz. We use the primitive implementations from the open software triNNity-benchmarks 1 (commit 8b0e642). This is executed from a Docker container. The benchmark is executed natively on the Intel and AMD machines. For the ARM system, primitives are compiled with cross-compilation. Primitives are profiled 25 times each to get the median value for reliability. In profiling, the layer configuration is run on a layer input with values drawn from a normal distribution. This is to simulate any possible image input to the network, although the execution time is not much affected by input image.

Preliminary Observations.
In either of the three datasets, no single primitive is routinely the quickest, neither is one family of primitives dominant in performance across all the layer configurations we tested with. Even when fixing the value for (the stride) and (the kernel size), the quickest primitives are spread across all families of primitives. Overall, the fastest primitives are spread across many of the tested primitives, as discussed in Section 3.1. For strided-convolutions, the im2 family often performs better, whereas, for unstrided convolutions, primitives from the kn2 family are often quicker. Whereas for unstrided convolutions with specific kernel sizes (one, three and five), families designed for the kernel size (conv-1x1, winograd3, and winograd5) often perform better.

Performance Modeling
We consider a linear regression model (Lin) as baseline and the two neural network models as previously discussed. The first neural network approach uses a separate fully-connected neural network for each primitive and data-layout transformation (NN1). The second approach uses only a single fully-connected neural network for all primitives and one for all data-layout transformations (NN2).
The profiler collected dataset is split into three subsets using the 80/10/10 split for training, validation and test respectively. The initial dataset is shuffled before split. Performance models are trained on the training set and hyper-parameters adjusted with observations from the validation set. To choose the neural network architecture size, we iterated by increasing the size until no further performance improvements is observed. For the per-primitive model (NN1), a single set of hyper-parameters is used across all models, fine-tuned for the best average performance across all primitives. Finally, the best performing model -as measured on the validation set -is tested on the test set to assess the final performance. These specific hyper-parameters we used for the final performance models are presented in Table 3.

Primitive Selection
We use the best performance model, determined as described in the previous section, to provide the costs for the primitive selection. To measure the primitive selection performance, we optimise the implementation of six neural networks: Alexnet [19], VGG-11 and VGG-19 [22], GoogLeNet [23], and ResNet-18 and ResNet-34 [10].
These are representative networks of current neural networks, with both classic ones (AlexNet) and more wide-spread ones (ResNet).
The six neural networks are optimized with primitive selection on both profiled execution time and on performance model estimations. The performance model is trained as described in the previous section. The BPQP-solver is used as an optimiser, and implemented as described in [9].

Transferability
We train each performance models using the performance measurements from just one platform, which makes them platform dependent. It is inconvenient to train the performance models for a new platform from scratch, due to the amount of time it takes to collect all the required training data. Instead, we apply a transfer learning approach to specialize the performance models to another platform with minimal amount of profiled sample points.
First, we evaluate the effect of applying the performance model trained on the Intel machine directly on the models optimised for the ARM and AMD machines compared to the optimal primitive selection for those target machines. By this, we run the primitive selection of Intel optimised networks on the other two platforms to see the gap in performance if models are not hardware dependent. This is tested on the layers of GoogLeNet due to its large variety in convolutional layers. A biggest degradation in performance may also be due to predictions of the Intel machine being, on average, significantly smaller due to the higher processor clock speed.
Second, we determine a scale by which the estimation of the Intel model can be calibrated to produce more comparable values to those measured on the ARM and AMD platforms. This scale factor is determined by using only 1% of the collected performance sample points on the two machines. Both 'factor-corrected' Intel models -one for the AMD and one for the ARM dataset -are then evaluated in the primitive selection process.
We know from previous work that by using transfer learning, the burden of collecting a large training set for the target devices is reduced [20]. Here we want to know how much data is actually enough for the fine-tuning stage to produce a quality performance model for a different hardware. We take the best Intel performance model as the starting point for transfer learning. For both AMD and ARM platforms, 6 separate fine-tuning stages are performed on fractions of the initial training sets. These are randomly selected at 0.1%, 1%, 2.5%, 5%, 10% and 25% of the training data available for each of those two devices. Their estimations are used for the same task as before, as part of the primitive selection process for GoogleNet. Picking different subsets of the data can give different results. To evaluate this effect, each model is trained 25 times using a different subset of the data (sampled uniformly at random). We then compare these results with a performance model trained from scratch -without transfer learning -using the same fractions of the data, which we expect to perform worse.
Finally, the transferability between primitive families is evaluated. This is done by fine-tuning the Intel performance model to the AMD platform using just data from an individual primitive family (such as im2) and evaluating the result on all primitive families separately. This process is then repeated for all seven primitive families.

RESULTS AND EVALUATION
This section presents and discusses the results of our experiments.
Here we assess the quality of our performance models, their impact on the primitive selection for CNN optimisation, and the quality and utility of transfer learning for specialising performance models to another target platforms.

Performance Model Estimation Accuracy
The evaluation of Lin, NN1 and NN2 performance models trained on the Intel dataset is presented in Figure 4, expressed in MdRAE over the test set. Both neural network models significantly outperform the linear regression model on nearly all primitives, justifying the need for non-linear models for performance prediction. The linear regression solution performs well for some of the primitives, most notably for the direct-sum2d and conv-1x1 primitive families. This can be explained by the lower complexity of those families of primitives compared to the rest, limited memory requirement and sequential access. Except for the winograd family, both neural network models predict all primitives with a MdRAE of around 2% (i.e. predictions are off by about 2% in the median values from the profiled execution time on the device for a given layer configuration). Most primitives in the winograd family perform with a MdRAE between 2% and 4%, while some primitives even reach errors as high as 10%. The reason why the winograd family may be harder to estimate is likely due to the fewer sample point in the training data for its complexity of performance values. Between the two neural network models, we see NN2 performing better than NN1 across all primitives. This is due to having access to alternative perspectives from all primitives and their performance to draw relations during training.
We choose the NN2 model as our default performance model to use in the following experiments since it has the lowest MdRAE out of the three models discussed here. We also validate this observation about our NN2 performance model on the other two platforms -on the AMD and ARM. Similar to the experiments for the Intel platform, we train the NN2 performance model on the profiled data from the AMD and ARM devices. The estimation error on their respective test sets is presented in Figure 5. We see the AMD trained model estimates the performance of primitives about as good as when trained for the Intel platform, just above 2% estimation MdRAE. Whereas for the ARM platform the error is in general between 4% and 6% for most primitives.
The MdRAE for estimating the execution time of data-layout transformations is presented in Figure 6. Transformations from and to the same layout are not presented as these can be skipped from execution (cost zero). We see the estimations with neural networks are very accurate, most errors being around 1% for the neural network models. Whereas using the linear regression model we observe very high errors. The NN1 models perform better for transformations from 'hwc', whereas the NN2 model performs better for transformations from 'chw' formats. Since NN2 has a relatively stable accuracy across all transformations, around 1% error, we choose this performance model solution as our default option.  In both cases (for primitive execution time estimation and for data layout transformation time) the NN2 models have a slight edge over the NN1 models. All the following experiments are conducted using the NN2 as our performance models (one for estimating primitive execution time and another one for estimating the execution time of data-layout transformations).

Primitive Selection
Selection Speed. Each of the two stages of primitive selection involves a time cost: the time required to measure or estimate the performance (execution time) of primitives/transformations and the time needed to apply the solver across the network layers (PBQP time). Table 4 shows the time required to perform the entire primitives selection on our selected CNNs. The biggest cost is the time required to profile all the convolutional layers by iterating over the primitives in order to generate the cost graph for the solver. We can see this is in the range of hours for the smaller platform (ARM), but also significantly larger for the Intel and AMD platforms (0.57 hours and 1.76 hours respectively) when profiling VGG-19. Performing the optimisation across the network with the PBQP solver incurs its own cost, which is included in the values presented in Table 4. But since this cost is unavoidable for both cases we mainly concentrate on reducing the first cost.
By using the performance model, the speed of estimating the cost of any primitive with any layer configuration is reduced to millisecond. This is presented in the second column of Table 4 for our chosen networks. These estimations are produced on the Intel platform. As observed, estimating the performance of primitives and transformations for the primitive selection of VGG-19 is 673ms, whereas profiling those on the ARM system takes 4.58 hours. This indicates a speedup of 25,000× to optimise the network for the ARM device with our performance model instead of profiling on the device.
Inference overheads. After seeing that our performance model can drastically speed-up the primitive selection process, we now analyze the quality of the solution in terms of its inference time for the optimised CNN (with the selected primitives on their estimated performance).
As we have seen, the estimated performance of primitives is very accurate, but the small errors might influence the quality of the identified solution with PBQP. Figure 7 shows the increase in  inference time of networks optimized by PBQP using the NN2 performance model, compared to models optimized using the profiled execution times (i.e. the approach first described in [1]). For all networks we considered, the execution time for the optimised network, with estimated performance of primitives, never increased by more than 1.1%, across all platforms, which is negligible. Overall, the inference time increase is generally smaller for the Intel platform, less than 0.7%, and larger for the ARM platform, but still below 1.1%. In fact, for the ARM platform, the estimated performance helps to find the optimal solution (0% execution time increased) for three of the networks: AlexNet, VGG-11 and VGG-19.

Transfer Learning
We first understand the need for transfer learning by observing the gap in performance when adopting the Intel performance model for an ARM and AMD machine -directly and with output scaling factors.
Factor correction. Figure 8a shows the MdRAE for the Intel and factor corrected Intel performance model (as described in section 4.4) when evaluated on the ARM and AMD test sets. As expected, applying the Intel performance model directly to a new platform results in a significant reduction in performance. The MdRAE goes as high as 820% when evaluated on the ARM dataset, much higher than the native model performance (4.6%), which is achieved when training the ARM performance mode on the training set of the device. Just by correcting the outputs of the Intel performance model with a factor (Factor Intel), however, seems to be good enough to adapt the model estimation by a factor to match that of a different target platform. This factor is determined independently for each primitive by using 25 sample points from the device (1% of the profiled dataset). By this approach, the MdRAE of the Intel performance model improves to 14% when evaluated on the ARM dataset. Figure 8b shows how using the Intel performance model influences the primitive selection process for a network running on the ARM and AMD platforms (both applied directly and with an output scaling factor). These are measured for optimising GoogLeNet with estimated primitive execution time, versus the actual profiled time on the ARM and AMD machines. Using uncorrected Intel performance model for the ARM platform (having a MdRAE of 820%) results in an inference time increase of just over 8%. Although this is significantly larger than the native performance (i.e. 1.1%) when using an ARM performance model, it would still be much quicker than an unoptimized network. With the factor correction, the inference time increase is half that of the increase when using the Intel performance model directly in optimising the GoogleNet for both ARM and AMD. We see there is still a gap between these performance estimators and what a NN2 native trained for the AMD and ARM can achieve. We set to close this gap by using transfer learning on the Intel trained performance model.
Transfer learning. Figure 9 (c) and (d) show how the Intel trained performance model (without factor correction) can be further improved by fine-tuning (i.e. continuing to train the model) using a small fraction of the training data available from profiling the target platform. By comparison, Figure 9 (a) and (b) present the accuracy of training the performance model from scratch on the same fraction of training data used for fine-tuning the Intel model. For both target platforms (i.e. AMD and ARM) 10% of our original training set is sufficient to train a performance model: • with errors of 7% and 8% for training from scratch and 5% and 5.7% for transfer learning (predictive performance). • and an inference time increase of 4% and 5.3% for training from scratch and 1.4% and 1.9% for transfer learning (primitive selection performance).
For transfer learning, these are close in performance to a performance model trained over the entire training set (dotted lines). The differences increase when lowering the amount of data available for training. The accuracy of the performance model trained from scratch degrades steeply when using just 1% of the original training data. The inference time increases by more than 20% when using the performance model trained from scratch. Whereas the transfer learning model performs much better, increasing the configured network inference time by only 4%. Figure 10 shows an even larger difference when using only 0.1% of the training data. At the other extreme, making more data available for training (more than 10%), benefits both the trained from scratch model and the transfer learning model. At 25% of data, our transfer learning performance model is only marginally worse (less than 1% error) than a model trained on all available data ( Figure 9).
Finally, table 5 provides further insight into how well data from one primitive family transfers to the other families. From this table, we can see, for example, that when the Intel performance model is fine-tuned with just data from the wino3 primitive family, it still performs reasonably well on wino5 primitives, but performs  Figure 9: The predictive and primitive selection performance of the ARM and AMD performance models, trained using various amounts of training data (shown on the x-axis). In the top two figures, the models are trained from scratch with the indicated amount of data from the original training set. In the bottom two, the Intel performance model was used as a starting point to fine-tune with that data. In all figures, the dotted lines shows the performance of the ARM and AMD performance models when trained from scratch using all the available training data.  Figure 10: The predictive and primitive selection performance of the ARM and AMD performance models, trained using 0.1% of the data. The performance of models trained from scratch is compared to models trained using transfer learning (using the Intel performance model as a starting point). This figure is an extension to fig. 9 (which shows the same comparison for different amounts of data).
badly on the im2 and kn2 family. A model trained with just data on the im2 primitive family, on the other hand, performs reasonably well on all other primitive families. Beyond learning from just one family of primitives, specialising the model even further requires some data from other families, but in lower amounts. As seen, the model already performs well when using just data from the im2 primitives. Supplementing that with a smalls number of data points from the other families can result in a model almost as good as the one trained on all our profiled training data.

CONCLUSIONS
We introduce primitive inference time modeling with a neural network as an efficient solution for removing the heavy profiling in primitive selection. Our approach reduces the time taken for obtaining the best configuration of a neural network from hours to seconds. This improvement enables any unseen CNN to quickly reach high performance on any given hardware platform. We show that the inference time of our configured networks increases by only 0.39% on average (worse case 1.1%) relative to the lengthy profiling approach in primitive selection. Our performance model is also viable for transfer learning, requiring only a small amount of profiling on a new platform to accurately adjust the pre-trained performance model. We lower the amount of training data required to construct a performance model from scratch to just 1% when using transfer learning, which results in a below 4% increase in inference time for our configured network.
Future Work. Here, we explored primitive performance modeling only on CPUs. The next step is to expand to performance modeling of primitives on GPUs. This will enable heterogneous CPU and GPU primitive selection. A 'data-transfer' cost (similar to that of data-layout transformations) will be used when transferring data between devices. Section 5.3 showed that transfer learning can vastly reduce the required amount of data. The particular subset of relevant training points needed in fine-tuning can be determined to minimize the required profiled configuration points, while maximizing the performance.