{'fraction_non_alphanumeric': 0.03835448675128034, 'fraction_numerical': 0.011467379202850144, 'mean_word_length': 5.166952740588168, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications

There are many deep learning (e.g., DNN) powered mobile and wearable applications today continuously and unobtrusively sensing the ambient surroundings to enhance all aspects of human lives. To enable robust and private mobile sensing, DNN tends to be deployed locally on the resource-constrained mobile devices via model compression. The current practice either hand-crafted DNN compression techniques, i.e., for optimizing DNN-relative performance (e.g., parameter size), or on-demand DNN compression methods, i.e., for optimizing hardware-dependent metrics (e.g., latency), cannot be locally online because they require offline retraining to ensure accuracy. Also, none of them have correlated their efforts with runtime adaptive compression to consider the dynamic nature of the deployment context of mobile applications. To address those challenges, we present AdaSpring, a context-adaptive and self-evolutionary DNN compression framework. It enables the runtime adaptive DNN compression locally online. Specifically, it presents the ensemble training of a retraining-free and self-evolutionary network to integrate multiple alternative DNN compression configurations (i.e., compressed architectures and weights). It then introduces the runtime search strategy to quickly search for the most suitable compression configurations and evolve the corresponding weights. With evaluation on five tasks across three platforms and a real-world case study, experiment outcomes show that AdaSpring obtains up to 3.1x latency reduction, 4.2 x energy efficiency improvement in DNNs, compared to hand-crafted compression techniques, while only incurring<= 6.2ms runtime-evolution latency.


INTRODUCTION
In recent years, a lot of ubiquitous devices (e.g. smartphones, wearables, and embedded facilities) are integrated with continuously running applications to facilitate all aspects of human lives. For example, the smartphonebased speech assistant (e.g. ProxiTalk [57]), and wearable sensor-enabled activity recognition (IMUTube [34], • Firstly, it is non-trivial to continually scale up/down the DNN compression configurations, including both architectures and weights, to meet the dynamic optimization objectives on multiple DNN performance (i.e., accuracy, latency, energy consumption) on-the-fly. This is because most DNN compression methods are irreversible to scale up DNN again, i.e., recover fine details of the DNN architecture and weight, from a compressed/pruned model. And the weight evolution is always limited by offline retraining. • Secondly, it is intractable to provide an efficient and effective solution to the runtime optimization problem.
To tailor this problem, quickly searching for the suitable compression techniques from an elite candidate set and efficiently evolving weights without model retraining are required. Moreover, it is difficult to systematically balance the compromising of multiple conflicting and interdependent performance metrics (e.g. latency, storage and energy efficiency) by merely tuning the DNN compression methods.
In view of those challenges and limitations, we present AdaSpring, a context-adaptive and runtime-evolutionary deep model compression framework. It continually controls the compromising of multiple performance metrics by re-selecting the proper DNN compression techniques. To formulate the dynamic context, we formulate the runtime tuning of compression techniques by a dynamic optimization problem (see Eq.(1) in § 3). In that, we model the dynamic context by a set of time-varying constraints (i.e., accuracy loss threshold, latency and storage budgets, and the relative importance of objectives). And then, we present a heuristic solution. In particular, to eliminate the runtime retraining cost, we decouple offline training from the online adaptation by putting weight tuning ahead in the training of a self-evolutionary network (see § 4). Furthermore, we present an efficient and effective search strategy. It involves an elite and flexible search space (see § 5.1), the progressive shortest candidate encoding, and the Runtime3C search algorithm (see § 5.2) to boost the locally online search efficiency and quality. The main contributions of this work are summarized as follows.
• To the best of our knowledge, AdaSpring is the first context-adaptive, and self-adaptive DNN compression framework to continually shrink model architectures and evolve the corresponding weights by automatically applying the proper retraining-free compression techniques on-the-fly. And it trains a self-evolutionary network to synergize the multi-scale compression operators' weight recycle and decouple offline training from online compression. • AdaSpring presents an efficient runtime search strategy to optimize the runtime adaptive compression problem. It introduces the elite and flexibly combined compression operator space, the fast Runtime3C search algorithm, and a set of speedup mechanisms to boost the search efficiency and quality, at runtime, while avoids explosive combination. • Using five mobile applications across three platforms and a real-world case study of DNN-powered sound recognition on NVIDIA Jetbot, extensive experiments showed the advantage of AdaSpring to continually optimize DNN configurations. It adaptively adjust the compression configurations to tune energy cost by 1.6 ∼ 5.6 , latency by 1.3 ∼ 10.2 , and storage by 201 ∼ 1.9 , with ≤ 2.1% accuracy loss. And the online evolution latency of compression configurations to meet dynamic contexts is ≤ 6.2 .
In the rest of the paper, we review the related work in § 2, present the system overview in § 3, and elaborate the AdaSpring design in § 5 and § 4. We report an evaluation of AdaSpring in § 6 and conclude in § 7.

RELATED WORK
Our work is inspired by and closely related to the following works.
DNN Compression for Ubiquitous Mobile Applications. There is a promising trend to bring deep learning powered intelligence to mobile and embedded devices (e.g. smartphones, wearables, IoT) for enhancing all aspects of human lives. such as smartphone-based speech assistant (e.g. ProxiTalk [57]), smarthome device-enabled sound detection [3], and wearable-based activity recognition (e.g. MITIER [8]). And recent research has demonstrated the potential of feeding DNNs into resource-constrained mobiles [12]   including parameter pruning [23], sharing [53], and quantification [63], compact component [24,25], and model distillation [7], However, we note that few efficient compression techniques are dedicated to optimizing the application-driven system performance (e.g. energy efficiency). For example, the recent work [27] argues that the platform-aware SqueezeNet [25] and SqueezeNext [18] merely reduce parameter size or MAC amount which do not necessarily lead to reduced energy cost or latency [56]. Also, all of these techniques need several epochs of model retraining to ensure accuracy, thereby they can not be locally online. Instead, AdaSpring decouples DNN training from online adaptive compression, and consider both the dynamic arithmetic intensity of parameters and activations for guiding the best specialization of convolutional compression configurations.
On-demand DNN Computation for Diverse User Demands. There have been two categories of on-demand DNN computation adjustment methods: on-demand DNN compression [42] [43] and on-demand DNN segmentation [60]. They specialize DNN computation offline to meet diverse hardware resource budgets (e.g. battery, memory, and computation) and application demands (e.g. input diversity). To satisfy resource budgets, He et al. [22] adopt reinforcement learning to adaptively sample the design space. Shuochao et al. [58] use a recurrent model to control the adaptive compression ratio of each layer. Singh et al. [48] introduce a min-max game to achieve maximum pruning with minimal accuracy drop. These methods, however, need extra offline training to update the meta-controller for on-demand adjustment. Zhao et al. [60] present the adaptively distributed execution of CNN-based applications on resource-constrained IoT edge clusters. However, because of mobile platforms' mobility and opportunistic connectivity, distributed DNN inference in mobile clusters is not robust yet. Built upon these efforts, AdaSpring is the first to enable runtime and adaptive DNN evolution locally without requiring Wi-Fi/cellular networks to connect with other platforms while achieve competitive performance.
Dynamic Adaptation of DNN Execution. Prior works have investigated the run-time adaptation of DNN execution to adapt to diverse inputs from two directions: dynamic selection of inference path [54] or network variant [50]. Wu et al. [54] adaptively choose which residual blocks to execute during inference to reduce computation without degrading accuracy. Teerapittayanon et al. [50] propose a multi-branch network to allow inference adaptively exit from early branches. Han et al. [20] adaptively select model variants to optimize accuracy and satisfy resource constraints (e.g. memory and energy). Gao et al. [17] propose feature boosting and suppression method to predictively amplify salient convolutional channels and skip unimportant ones. However, these methods highly depend on the pre-defined design space of alternative execution paths and variants, but it is prohibitive to specify all of them before deploying models into agnostic mobile contexts. AdaSpring dynamically select and combine the proper compression operators to flexibly shrink the model configurations from multiple scaling dimensions at runtime.
Fast and Platform-aware Neural Architecture Search. Recent studies have verified the potential of leveraging neural architecture search (NAS) framework to automate neural architecture specialization for mobile applications, from two aspects. Firstly, the Fast-NAS aims to automatically specialize neural architecture for different performance demands (e.g. computation amount, parameter size), using as little search cost as possible [44]. To speedup the search, researchers have investigated the modular search strategy [6,39,61], differentiable search strategy [11,28,40,62], and super-network search strategy [1,5,15]. For example, Liu et al. [40] relax the search space to be continuous, so that it can be optimized by gradient descent, using orders of magnitude less search cost. Cai et al. [5] trains an once-for-all (OFA) super-network that supports 2 × 10 19 diverse variant-network search. We note that the OFA super-network includes some redundant and invalid variant-networks, which is not elite and incurs a high search cost. (As we will discuss in § 5.) Secondly, unlike general NAS that only optimize for model-relative metrics, such as FLOPS, the platform-aware NAS alao incorporates platform-relative metrics (e.g. latency) into optimization objectives. Such as Mingxing et al. [49] explicitly incorporate latency into the NAS objective to identify a mobile CNN model. Xiaoliang et al. [13] propose an efficient search algorithm aided by efficient accuracy and hardware resource predictors. However, above methods still sacrifice high overhead to obtain the ranking of candidate architectures based on their performance on validation sets. And they donot accurately consider the energy consumption improvement target since the energy efficiency measurement is not straightforward on different platforms with dynamic nature. Depart from existing efforts, AdaSpring treats the retraining-free compression operator (illustrated in § 5.1) as a new ensemble to be tuned by automated macro-NAS. It trains a self-evolutionary network(see § 4) at design time to decouple model retraining and adaptive compression. Besides, it present a set of mechanisms to boost search efficiency (i.e., at millisecond level) and quality during dynamic inference. Notably, AdaSpring leverages the dynamically measured hardware-relative metrics (i.e., arithmetic intensity of parameter and activation) to derive a guiding selection, which also prevent the explosive combination (see § 5).

OVERVIEW
This section starts with problem analysis and then presents an overview of AdaSpring design.

Problem Study
Due to the dynamic natures of DNN deployment context, we aim to continually tune the DNN compression configurations to directly/indirectly optimize the application-driven system performance (i.e., accuracy, energy efficiency, latency). The hybrid dependency of multiple platform-relative performance metrics and DNN-dependent metrics are shown in Figure 3. To further understand the performance requirements of DNN for the continuously running mobile applications, we ask 60 mobile users and 10 Android developers to rate the importance of different DNN performance aspects on mobiles. And we summarize the results as our design goals. Specifically, a DNN for continuously running mobile Apps needs to fulfill the following requirements: • Accurate: the DNN is accurate enough to guarantee a high-quality task. The model weights at different scales are well-trained to represent the generic information of recognition objects. • Responsive: the complexity of the DNN should be controllable to satisfy diverse user demands on latency constraints, especially on low-end (e.g. CPU-powered) mobiles. • Energy-Efficient: the energy consumption of the DNN should be continually optimized, which is the bottleneck metric for continuously sensing applications [56].  • Runtime-evolutionary: both the DNN architecture and parameter weights are runtime-evolutionary to meet the dynamic deployment context for continually optimizing the above three requirements (i.e., accurate, responsive, energy-efficient) at runtime. Unfortunately, none of previous efforts satisfy all these requirements (as discussed in § 2). To this end, this paper proposes AdaSpring, a context-adaptive and runtime-evolutionary DNN compression framework to automatically optimize the requirements mentioned above, which are closely related to the user experience.

Optimization Formulation
As shown in Figure 3, AdaSpring intends to provide a systematic method to automatically select the compression operator combination for tuning the above conflicting and interdependent performance metrics. Mathematically, AdaSpring explores an efficient solution to the following dynamic optimization problem: where ∆ represents the set of all optional convolutional compression operators (as enumerated in § 5.1). Given an backbone-net architecture Ω, Ω( ) represents the re-configured model architecture compressed by the selected compression operator . , , and denote the measured accuracy, energy efficiency, latency, and memory footprint of a given model running on the target mobile platform. The two objectives on and are combined by relative importance coefficients 1 and 2 , which dynamically depend on the platform's remaining battery. We express the dynamic deployment contexts as a set of time-varying constraints, i.e., the threshold of accuracy loss ℎ ℎ ( ), the latency budget ( ), the storage budget ( ), and relative importance coefficients of objectives ( 1 ( ), 2 ( )). The latency budget ( ) is application-specified. And the storage budget (t) is platform-imposed. For example, reducing the model size to satisfy the budget of L2-Cache helps to fit it  into the on-chip memory and avoids the expensive off-chip access. We note that (.) is a normalization operation for objective aggregation, e.g.
(.). We then propose a heuristic optimization solution as adjusting the model architecture for satisfying dynamic performance requirements. In particular, the model architecture Ω( ) can directly determine both and (see Figure 3). While the quantification of hardware-dependent metrics and are not straightforward. Therefore, AdaSpring's goal turns to adaptively select the compression operator combination from a discrete set of all possible combinations ∆, so that it can directly/indirectly tune model performance metrics.

AdaSpring Framework
The above challenging problem motivates the AdaSpring design. As shown in Figure 4, the AdaSpring framework consists of a self-evolutionary network, a runtime adaptive compression block, and a dynamic context awareness block. (i) The self-evolutionary network is an ensemble of a backbone-net and multiple retraining-free compression operator-variants, which enables weight recycle between numerous variants while avoiding catastrophic interference. We initialize the backbone-net's hyperparameters at design time using an on-demand DNN generation framework, i.e., AdaDeep [41], for satisfying mobile application performance demands on a target platform. (ii) The runtime adaptive compression block is capable of selecting a deterministic optimal combination of compression operators for reconfiguring and evolving the backbone-net at runtime. And (iii) the dynamic deployment context awareness block detects the evolution demands and triggers the runtime adaptive compression block. The triggering station can be modeled as the noticeable context changes or by a pre-defined frequency (e.g. time slice) for continuously running Apps in regular days.

RETRAINING-FREE AND SELF-EVOLUTIONARY NETWORK DESIGN
This section presents the design of the retraining-free and self-evolutionary network. The self-evolutionary network consists of a high-performance backbone network and multiple compression operator-variants.

Compression Operators
This paper focus on the configuration optimization of convolutional architecture, operations, and activations. Because recent successful DNN models tend to shift more parameters on convolutional layers and use fewer fully-connected layers [5,10,53]. Built upon the existing compression experience, we propose the following alternative convolutional compression operators that synthesize multiple scaling dimensions (e.g. width, depth, and connection).
• Compression operator 1 : multi-branch channel merging techniques(e.g. Fire block [25]) increase the model depth with less parameters by replacing one conv layer using two conv layers (i.e., squeeze layer and expand layer) which is elaborately designed to decrease the kernel size and channel size per unit.

Ensemble Training of Self-evolutionary Network
We put the retraining process ahead in the ensemble training of the self-evolutionary network at design time to get rid of weight retraining during dynamic inference. Therefore, the self-evolutionary network training is an ensemble of a backbone-net and multiple variant-nets derived by various convolutional compression operators.

Primer on Parameter
Recycling. We refer to the parameter recycling strategy [4,53] to recycle the backbonenet weights and take less search time than those searching from scratch. The weight recycling strategy is conducive to making maximum use of the existing architectures' experience to reduce the time complexity of the searching process. In particular, we reuse the existing hand-crafted/elaborated high-performance DNN, including architecture and weight, as an initialization point (i.e., backbone network). And then, we leverage an automated optimizer (i.e., search strategy) to only search for the optimal architecture adjustments (e.g. widening a certain network, skipping connections) to obtain a promising new model. However, these methods challenge the ensemble training of multiple variant models that drift away from the backbone-net's initial configuration. In detail, the training of a variant's weights will likely interfere/override the weights that have been learned for other variants and thus degrade the overall performance. We note that the catastrophic interference problem when multiple variant-nets share parameters is a long-standing problem in itself [16,30].

Training Strategy.
Our goal with the self-evolutionary network is to integrate with multiple versions of DNN architectures and the corresponding weights introduced by different compression operators. And the above-mentioned parameter recycling strategy provides great potentialities. To further avoid the catastrophic interference problem caused by parameter recycling, we present a novel training strategy to consider the parameter transformation and knowledge distillation for preserving the parametric function of multiple variant-nets. In detail, we first perform the standard back-propagating process to train a high-accuracy backbone-net. Afterwards, we respectively leverage the parameter transformation techniques for learning compression operators 1 and 2 , the knowledge distillation techniques for learning compression operators 3 and 4 , and the trainable channel-wise mutation techniques for learning 3 .
(1) Parameter transformations for learning variant-nets derived by compression operator 1 and 2 . We consider the function-preserving parameter transformation when recycling parameters. It allows us to initialize a new variant-net that is derived by a compression operator to preserve the function of the given backbone-net, but use different parameterization to be further trained to improve the performance [4]. We transform the original convolutional parameter and store the extra copy of weight for 1 and 2 . And we further set an accuracy target as a threshold, by which the transformed parameters for compression operator-variants will only be fine-tuned when its accuracy is lower than that. As thus, we need a small number of extra parameters to store the transformed parameters for compression operators 1 and 2 . And we only access the weights of the deterministically selected compression operator to evolve model weights.
(2) Knowledge distillation for learning variant-nets derived by compression operator 3 and 4 . We allow each conv layer to choose the depth/channel compression ratio flexibly. And we adopt the knowledge distillation techniques [5] to fine-tune the parameters of compression operators 3 and 4 with different compression ratios (e.g. 20%, 50%). So that AdaSpring can flexibly switch over the different parameter weights of channel-wise and depth-wise scaling operators, and avoid weight interference. Also, we perform the trainable channel-wise and depth-wise architecture ranking as the weight importance criterion to guide the adaptive layer slimming and scaling. In particular, we pre-train a self-evolutionary network to evaluate the overall performance (e.g. accuracy drop, parameter arithmetic intensity, activation arithmetic intensity, and latency) of different variant-networks that are compressed by different operators. And these are used as the prior-based architecture importance ranking to guide the runtime scaling to shrink unimportant layer/channel first, rather than randomly scaling. (3) Trainable channel-wise mutation for training variant-nets derived by compression operator 3 . To maintain a good diversity of solutions, we present a novel trainable architecture mutation technique to inject the architecture variance into the compressed network. This idea is supported by recent DNN studies, which have verified the dominant effect of model architecture on accuracy compared to model parameters [59]. That is, AdaSpring can directly use the trainable architecture mutation technique with diverse noise magnitude since the channel importance ranking of the backbone-net is consistent at both design time and runtime. Specifically, we inject Gaussian noise to the channel-wise operator's scaling ratio (i.e., 3 ), and the noise magnitude is trainable for channel importance ranking. That is, the more important the channel is, the lower intensity of noise we inject. This, as we will evaluate in § 6.5, plays a nontrivial role for AdaSpring's progressive shortest encoding process of DNN compression configurations and the runtime searching process for boosting the runtime DNN adaptation quality and efficiency. Besides, to enable the stable ensemble training of multiple variant-nets, we leverage the mini-batch techniques to split the training data into small batches. We normalize the gradient to reduce the interference caused by gradient variance [38].

RUNTIME ADAPTIVE COMPRESSION
This section presents how AdaSpring quickly searches for the most suitable combination of retraining-free compression operators, from a flexible and elite space, to reconfigure the trained self-evolutionary network on-the-fly.

Flexible and Elite Search Space
We form an elite search space, which include a set of coarse-grained compression operators (e.g. Fire block [25], SVD-based [53], sparse coding-based [2] factorization), for faster convergence, and the fine-grained compression operators (e.g. channel-level and depth-level pruning and channel-wise randomization), for better diversity. Consider a convolutional layer that has the total parameters: (input feature map channel size ) × (output feature map channel size ) × (kernel width/height )× (kernel width/height ), and total activations: × (output feature map width/height) × . Scaling either the input feature map, kernel, channel, or output feature map can shrink the model complexity. We empirically observe that different scaling dimensions are not independent. Firstly, there is no single compression technique that achieves the best application-driven performance (i.e., , , , , , and ). It is necessary to combine several compression techniques. Secondly, as mentioned in § 2, few existing compression techniques are retraining-free or dedicated to optimizing the holistic hardware efficiency across various platforms. These findings further suggest us toflexibly coordinate and balance multiple scaling dimensions by searching for the best combination of compression operators, rather than the single dimension (e.g. model pruning).

Hardware Efficiency-guided Combination.
We argue that the widely used parameters number, MAC amount, or speedup ratio are not good approximations for hardware efficiency, which heavily depends on the memory movement and bandwidth bound. For example, Jha et al. [27] reported that although SqueezeNet [25] has 51. fewer parameters than AlexNet [33], it consumes 33% more energy due to its larger amount of activations and data movement. And we identify that merely cutting down the parameter size may lead to an increase in activation size, which, in turn, increases the memory footprint and energy consumption [26]. For example, the recent study [26,27] has shown that the energy consumption of CNNs mainly depends on the memory movement, memory reuse, and bandwidth bound.
To this end, we present the controllable hardware-efficiency criteria, i.e., arithmetic intensity, to guide the automated combination of compression operators in different layers. We leverage the arithmetic intensity as a proxy to the degree of reuse of parameters and activations and the energy consumption required for processing inputs, inspired by hardware studies [26,27]. Because the measurement of hardware-relative metrics, especially energy efficiency, is not straightforward. Thereby, we present three hardware-efficiency metrics to predict how efficiently arithmetic operation can reuse the data fetched from different levels in the memory hierarchy and how efficiently the arithmetic operation is executed.
• Computation/parameter ratio / : is an approximation of the parameter arithmetic intensity; • Computation/Activation ratio / : is the proxy of the activation arithmetic intensity; • latency : include the measured inference time of a specialized model, and the time i.e., for loading parameters and activations for convolution computing on the target mobile device, i.e., = + . We separately evaluate / and / and then aggregate them together by the aggregation coefficients 1 and 2 , to better profile the energy efficiency of each candidate compression operator .
Upon these criterions, AdaSpring automatically selects and combines compression operators for maximizing the aggregated value of 1 / + / , according to the upper limit of the calculation intensity of the mobile platforms. And test the real latency to prevent the exploration of invalid solutions via comparing with the latency budgets. We empirically set > (e.g. = 0.4, = 0.6 as default) since / contributes more to memory footprint (as benched in § 6.5). And AdaSpring discovers some novel combinations for optimizing the underlying data movement. For example, we suggest the 1 + 3 and 2 + 4 groups (as discussed in § 6). The fine-grained channel-wise scaling operators (e.g. 3 , 4 ) readjust the channel size, MAC amount, and output activation size of the conv layers to smooth out the bandwidth bound problem, which is caused by the coarse-grained operators  (e.g. 1 and 2 ). The hardware efficiency-guided combination of several compression operators also helps to avoid the blindly explosive combination.

Runtime Search Strategy
To evolve DNN architecture and weight to an optimal configuration at runtime, we propose the runtime search strategy based on the above flexible and elite search space.

Progressive Shortest Encoding of Candidate.
Consider a complex self-evolutionary network that contains many combinations of compression operator variants and configurations, systematically and generically choosing the right candidate configurations and encoding them into the representation of a search algorithm is difficult. For optimizing DNN compression configurations at runtime, such representations define the potential search space of the problem to be explored. Given that some candidate configurations do not contribute to the specific performance optimization demand or other candidates can represent some of their information, the shortest actual encoding will benefit the search result (i.e., model evolution plans) and overhead. As shown in Figure 7(a), the classic binary encoding of all compression operator configurations across all layers in a binary format is redundant. Specifically, given a backbone network with conv layers to be selectively compressed. Take = 3 as an example. A classic binary encoding method needs 3 bit to record whether a specific layer participates in compression or not. Other 3 * 4 bits (four bits to represent 4 2 selective operators per layer). In this way, the encoding length is + = ( + 1) when we have optional compression operator. And the search space derived by this encoding diversity is 2 × , i.e., ( ). Furthermore, it will increase exponentially as the number of optional compression operators increase.
To better represent the fundamental search space, we propose the progressive shortest encoding of compression operator configurations via a layer-dependent manner. As we will show in § 6.5.3, it improves the search efficiency by one order of magnitude, compared to the classic binary encoding. As shown in Figure 7(b), we use digits to record the count of layers that have been compressed. The first digit represents the compressed layer count, and the next length-variable few digits record the selected compression operator index of each layer. For example, the value 1 of the first digit means that only the first conv layer is compressed on-demand. Thereby, only one additional digit is needed to record the compression operator index (i.e.,, 1) for it. Afterward, AdaSpring inherits the above 2-digit string and inject channel-wise variance to mutate the inherited survival 2-digit encoding string. We refer the channel-wise variance mutation process in § 5.2. And then, we turn to the second adaptable conv layer. If the first digit of compressed layer count is updated to 2, we append one more digit indicating the selected Mutate and augment 2 candidates to 6 candidates using the trained channel-wise variances ; 6 Select the Pareto-optimal candidate (i.e., min while max ) as the survival candidate ( ) ; compression operator index to the survival 2-digit encoding string. Thus, the encoding length progressively increases from 2 to ( + 1), and the complexity of the search space is reduced to ( 2 ). The progressive shortest encoding of the candidate is conducive to the flexibility of AdaSpring and prevents unnecessary exploration.

Runtime3C Search
Algorithm. This subsection presents the Runtime3C search algorithm, a Pareto optimal decision-based searching algorithm, to pick a sole optimal solution from the search space at runtime. To the best of our knowledge, many widely used universal search algorithms (e.g. evolutionary algorithms) are not designed to optimize the runtime adaptive compression problem or handle dependency constraints of multiple DNN performance. We heuristically regard the selection of compression operators for each layer as a single-layer optimization subproblem in a collaborative manner to derive the most suitable solution quickly and effectively. As shown in Algorithm 1, each subproblem at layer is to search the optimal group of compression operators for optimizing the overall performance of the entire DNN. Starting from the second conv layer by default, AdaSpring selects two candidate solutions at layer from the Pareto front of the selectable compression operator groups for optimizing the accuracy and energy efficiency of the entire model (line 2). In detail, the picked two candidate solutions are the best two compromises in 1 ( ) v.s. 2 ( ), from the Pareto front within the valid search space, i.e., > 5%. Here, we leverage the ranking of the pre-tested accuracy and energy cost of the DNNs to establish the Pareto front. And the accuracy ranking derived by historical results is consistent with the ranking of the actual accuracy of these DNNs measured on mobile devices. We then mutate and augment candidates from two to six by injecting the channel-wise variance to the candidate configurations. The trained architecture importance is a criterion for Gaussian noise injection. This process can improve the diversity of subproblem Image (5 classes) ImageNet [14] 65, 000 images 3 Acoustic event (9 classes) UbiSound [47] 7, 500 audio clips 4 Human activity (7 classes) Har [51] 10, 000 records of accelerometer and gyroscope 5 Driver behavior (10 classes) StateFarm [29] 22, 424 images solution as well as the performance of the global solution, inspired by the genetic algorithm in the adaptive software engineering [9]. We choose the best candidate as the survival subproblem solution for compressing layer (line 6). Afterward, the −th layer's survival solution is used to reconfigure the −layer and becomes the initial station of the subproblem at ( + 1)-th layer. We fix the selected compression configurations for −th layer and repeat the above-searching steps (line 3 ∼ 9) to specialize the optimal compression strategies for the ( + 1)-th layer. Once the model satisfies the dynamic constraints in latency ( ) and memory ( ) at time , the subproblem expansion stops (line 12). And finally, it outputs the global compression configuration solution.

EVALUATION
This section presents the evaluation of AdaSpring over different mobile applications on diverse mobile and embedded platforms with dynamic deployment context. We compare AdaSpring against ten alternative methods reported in the state-of-the-art literature.

Experiment Setup
We first present the settings for our evaluation.
System Implementation. We implement AdaSpring's offline block with TensorFlow [19] in Python on the server side to train the self-evolutionary network (see § 4). And we realize the AdaSpring's online blocks on the mobile and embedded platforms to adjust the DNN configurations on the fly for better inference performance. The self-evolutionary network (i.e., a backbone-net and multiple variant compression operators), generated by AdaSpring's offline component, is then loaded into the target platform. To further reduce the memory access cost, we load DNN parameters from L2-Cache memory.
Mobile Platforms with Dynamic Context Settings. We evaluate AdaSpring on three categories of commonly used mobile and embedded platforms, including one personal smartphones, i.e., Xiaomi RedMi 3S (device1), one embedded development board, i.e., raspberry Pi 4B (device3), and one mobile robot platform i.e., NVIDIA Jetbot (device4) loaded with the mobile development board. They are equipped with diverse processors, storage and battery capacity. The dynamic context is formulated by the time-varying latency budget ( ), storage budget ( ), and the relative importance coefficient of accuracy and energy efficiency objectives.
Comparison Baselines. We employ three categories of DNN specialization baselines to evaluate AdaSpring. The detailed settings of ten baselines from three categories are as below. Firstly, the hand-crafted compression baselines relay on manual design to realize efficient DNN compression. They provide the high standard for AdaSpring to tune the specialized DNNs' performance tradeoff between accuracy, latency, and resource efficiency.
• Fire [25] presented in SqueezeNet reduces filter size and decreases input channels using squeeze layers.
• MobileNetV2 [46] replaces the traditional convolutional operation by an inverted residual with the linear bottleneck to expand module to high dimension and then filter with a depth-wise convolution. • SVD-based convolutional decomposition technique [35] introduces an extra conv layer between and ( +1) using the singular value decomposition (SVD) based parameter matrix factorization. The number of neurons in the inserted layer is set according to the dynamic neuron numbers in , i.e., = /12. • Sparse coding-based convolutional decomposition technique [2] insert a conv layer between and ( +1) using the sparse coding-based parameter matrix factorization. The k-basis dictionary is dynamically determined by the neuron number in , i.e., = /6.
Secondly, the on-demand DNN compression baseline methods adopt a trainable optimizer to automatically find the most suitable DNN compression strategies for various mobile platforms.These baselines provide a strict benchmark against which we can validate that both searching and retraining costs are bottleneck limitations for the runtime adaptation demands.
• AdaDeep [41] automatically selects and combines compression techniques to generate a specialized DNN that balance accuracy and resource constraints. • ProxylessNAS [6] directly learns architectures without any proxy while still allowing a large candidate set and removing the restriction of repeating blocks. • Once-for-all(OFA) [5] obtains a specialized sub-network by selecting from the once-for-all network that supports diverse architectural settings without additional training.
Thirdly, the runtime adaptive DNN compression requires to search for the most suitable combination of retrainingfree compression techniques quickly, we select two baseline optimization methods to compare with AdaSpring.
Here, the baseline optimizers represent two intuitive searching ideas for the runtime adaptive compression of DNN configurations.
• Exhaustive optimizer tests all combinations of compression operators' performance on the validation and then selects the one variety with the best tradeoff based on the fixed performance ranking. And then it fixes the compression operators and only scale down the compression operators' hyperparameters, i.e., compression ratio, to satisfy the dynamic resource budgets. • Greedy optimizer selects the best compression operator layer-by-layer that obtains the best tradeoff between accuracy and parameter size, in which the relative importance is equally set to a fixed value of 0.5. • AdaSpring selects and applies the most suitable combination of compression operators into the selfevolutionary backbone network for accuracy and resource efficiency tradeoff.

Performance Comparison
We evaluate AdaSpring in terms of the specialized DNNs' running performance (i.e., accuracy , amount of MACs , parameter arithmetic intensity / , activation arithmetic intensity / , and energy consumption ) and the specialization methods' all-around performance (i.e., search cost, retraining cost, and scaling flexibility). As shown in table 2, we compare AdaSpring's performance with ten baselines. In this thread of experiments, we leverage the same mobile sensing task (i.e., image recognition using CIFAR-100 ( 1 ) datasets) and target embedded platform (i.e., Raspberry Pi 4B) for six state-of-the-art baselines and AdaSpring for a fair comparison. We adopt different baseline methods to specialize the DNN architectures and weights for optimizing accuracy and resource efficiency objectives with dynamically specified constraints (see § Equ. 1). Here, the relative importance coefficients (i.e., 1 and 2 ) are dynamically determined by the remaining battery percentage of the target platform, i.e., 2 = {0.3, }, and 1 = 1 − 2 . Afterwards, we test the specialized DNN's running performance on a Raspberry Pi 4B platform. To mitigate the effect of noise and increase the robustness of performance measurements, we repeat the steps mentioned above five times and take an average over them.
Performance comparison. Table 2 summarizes the performance comparison between ten baseline methods and AdaSpring. First, AdaSpring achieves the best overall performance in terms of accuracy , MAC amount , parameter arithmetic intensity / , activation arithmetic intensity / , and energy consumption , while incurring negligible accuracy loss, compared to the DNNs specialized by other baseline methods. The AdaSpring reduces the model inference latency to 15.6ms, the energy consumption to 1.9 . And it increases the parameter arithmetic intensity / to 158.9, the activation arithmetic intensity / to 358.7. Notably, AdaSpring generates DNN to get the largest activation arithmetic intensity / and second-largest parameter arithmetic intensity . Compared to the parameter size, the influence of activation arithmetic intensity upon energy consumption is equally or even more critical. The DNN specialized by the hand-crafted Fire, MobileNetV2, SVD, and sparse coding techniques consumes energy by 3.1 , 5.2 , 4.8 , and 4.6 , respectively. The accuracy of exhaustive optimizer is much lower than the proposed design, since it shows low accuracy when it fixes the compression operator categories and only over-compresses their hyperparameters. This outcome demonstrates that the reselection of different compression operators are necessary. The specialized DNN's accuracy achieved by AdaSpring is at least as good as ProxylessNAS, and sometimes even better than the hand-crafted compression techniques. Second, the AdaSpring's specialization scheme is the most efficient in reducing the searching cost 9 : 00 10 : 00 11 : 00 12 : 00  to search per adaptation, while AdaSpring only needs 3.8 to do that. This is because AdaSpring leverages the elite compression operator space, rather than the basic kernel size or channel number space in OFA, to avoid the redundant search exploration.
Summary. AdaSpring outperforms the other ten baselines in terms of the DNN performance tradeoff between accuracy, latency, arithmetic intensity of parameters and activations, and energy consumption. Meanwhile, it incurs the modest searching cost without retraining, making it ideal for runtime adaptive DNN compression.

AdaSpring's Performance over Different Tasks
To illustrate the AdaSpring's performance over different tasks, we evaluate it using all the five applications/datasets (see § 6.1) on a Raspberry Pi 4B platforms (Device 3) which is powered by a mobile 3800 ℎ battery. AdaSpring dynamically detects the platform's remaining battery and sets the coefficients between accuracy and energy efficiency in Equ. 1 according to the percentage of remaining power , i.e., 2 = {0.3, 1 − }. In addition, we specify the storage budget as 2MB that is capacity of the L2-Cache. We set a accuracy loss threshold to be 0.5, 0.3, 0.6, 0.5 for image classification tasks ( 1 , 2 ), sound sensing (i.e., 3 ), human activity prediction task ( 4 ), and driver behavior recognition task ( 5 ), respectively. And assume the latency sensitivity as the latency budget of 20 , 10 , 30 , 20 for 1 ∼ 5 .
Performance. Figure 8 compares the performance of the DNN configurations specialized by AdaSpring on five different tasks in terms of user experience metrics (i.e., inference accuracy , energy efficiency , and inference latency ) and direct DNN metrics (i.e., computation , parameter size , activation size ). And we compute the mean and standard deviation of the running performance of the DNN specialized in five dynamic moments, at which the percentage of remaining battery is 0.85, 0.75, 0.62, 0.52, and 0.38, respectively. These affect the tradeoff demands on objectives. The storage budget for parameters dynamically depends on the available Cache capacity. We simulate the unpredictable resource contention by other software using the randomization noise injection to Cache's available capacity, i.e., (2 − )MB. For different tasks, datsets, and deployment contexts, AdaSpring selects the various combinations of compression operators to scale up/down the model configurations to optimize and balance multiple performances. It achieves the inference latency 1.2 ∼ 2.8, the parameter arithmetic intensity 106 ∼ 229, activation arithmetic intensity 143 ∼ 220, with a negligible accuracy loss (≤ 0.5% ) or even accuracy improvement (≤ 2.2%).
Summary. For different tasks with diverse backbone model shapes and various sensitivity to accuracy loss and latency, the DNN specialized by AdaSpring varies. As for the same task, the DNN's compression configurations founded by AdaSpring also differ according to the dynamic deployment context.

AdaSpring's Performance across Diverse and Dynamic Deployment Contexts
In this experiment, we compare the AdaSpring's performance for mobile sound sensing application ( 3 ), tested in three different platforms. We adopt the same self-evolutionary network comprising of the same backbone-net and some optional compression operator-variants. Different platforms have different resource characteristics, which are further affected by dynamic deployment contexts. In particular, we adopt the RedMi 3S smartphone equipped with Qualcomm B21 processor, 2 L2-Cache, and 4100 ℎ battery; the Raspberry Pi 4B with 2 L2-Cache, and 3800 ℎ battery; and the NVIDIA Jetbot with quad-core ARM Cortex-A57 processor, 2 L2-Cache, and 7200 ℎ battery. We adopt the similar dynamic deployment context settings with § 6.3.
Performance. Figure 9 summarizes the performance of DNN specialized by AdaSpring along with the dynamic changes of deployment context. We first initialize the different DNN configurations for various platform constraints and then leverage AdaSpring to update the compression operator-variant selections according to the specific platform's dynamic contexts. We select four points of dynamic contexts. AdaSpring identifies DNN configurations to obtain latency of 1.1 ∼ 1.8, parameter arithmetic intensity 81 ∼ 151, and activation arithmetic intensity 192 ∼ 397 while have slightly degraded or even better accuracy 91.2% ∼ 98.6%. We pick a time fragment to show four moments with dynamic deployment contexts, as shown in Table 4. As the gradual reduction of battery power and the dynamic fluctuation of Cache capacity, we show the performance changes of the DNN, which is continually scaled-down/up by selecting and combining different compression operators. Moreover, AdaSpring supports scale up the model architecture again when the dynamic constraints on resource efficiency are relaxed, bringing better flexibility. Summary. AdaSpring adaptively selects the proper combination of compression operators to optimize DNN performance continually to meet dynamic context demands. Moreover, AdaSpring accomplishes a flexible evolution, i.e., support both scaling up and down the DNN configurations as the context demands change.

Micro-benchmarks of AdaSpring
In this subsection, we evaluate the impact of different factors on AdaSpring's design.
6.5.1 Hardware Efficiency-guided Combination. We compare the performance of DNNs reconfigured by a standalone compression technique (e.g. the Fire module [25]), the blindly combined two compression techniques (e.g. Fire module plus depth-wise pruning), and the proposed hardware-efficient grouping of compression operators (see Figure 10(a)). And we show that the hardware-efficient grouping can always guarantee a comparable overall performance in terms of accuracy, energy efficiency and latency.
6.5.2 Layer-dependent Inheriting and Mutation. . As discussed in § 5.2, we leverage the inheriting and mutation schemes to balance the searching diversity and convergence. We compare the locally greedy scheme layer by layer, the layer-dependent inheriting scheme, and the proposed layer-dependent inheriting plus mutation scheme in AdaSpring. Figure 10(b) shows that AdaSpring achieves the best tradeoff between model accuracy and energy efficiency. and progressive shortest encoding scheme. And we find that AdaSpring's progressive shortest encoding method boosts the search efficiency.
6.5.4 Aggregation Coefficients in Arithmetic Intensity. As mentioned in § 5.1.2, the aggregation coefficients 1 and 2 for the parameter and activation arithmetic intensity 1 / + 2 / need to be optimized empirically. Figure  10(d) illustrates the estimated energy consumption using different aggregation coefficient settings. Therefore, we set 1 = 0.4, 2 = 0.6 by default across different platforms.

Case Study
We deploy AdaSpring on a commercial mobile robot platform (i.e., NVIDIA Jetbot, device4) and conduct a one-day experiment (09:00 to 17:00) to continually optimize the DNN configurations for a sound assistant application (i.e., UbiEar [47]). This application adopts a DNN to realize a sound recognition and notification tool for hard-ofhearing people to sense emergency (e.g. fire alarms, smoke alarms, kettle boiling whistle) and social events (e.g. doorbell ring, knocking door, people crying). We simulate the dynamic mobile context of the DNN (as described in § 3) as follows. On the one hand, we artificially play some audio clips for emergency events and generate social events to control the happening frequency of acoustic events, affecting the DNN inference frequency. On the other hand, we simulate the unpredictable storage resource contention by other software using the randomization noise (e.g. Gaussian noise) injection to the available capacity of L2-Cache, i.e., (2 − )MB. Here, the maximum capacity of L2-Cache on NVIDIA Jetbot platform is 2 , and we update the randomized resource contention value of per hour. We do not artificially change the battery power, which is continuously consumed in the real-world as the application runs. Therefore, the remaining battery is dynamically changing, e.g., 86%, 72%, and 63%, as shown in Figure 13, which forms the dynamic energy budgets. Figure 13 illustrates the dynamic deployment context (i.e., energy, storage, event happening frequency) of the DNN for the continuous sound sensing application. The battery's remaining energy formulate the importance coefficient 2 in the runtime optimization problem (Equ. 1). The available capacity of L2-Cache decides the storage budget of parameters . And the sound emergency frequency will indirectly influence the battery's power. Different deployment contexts have various resource constraints and performance objective sensitivity, which lead to further performance and budget demands on the DNN. AdaSpring triggers the runtime DNN evolution block by a pre-defined frequency (e.g. every two hours) to shrink the DNN configurations in this regular day. Figure 12shows the runing performance of the DNNs specialized by AdaSpring. AdaSpring can continually and adaptively select the best compression strategy to shrink the DNN configurations given diverse user demands. Specifically, it selects the 1 (Fire) + 3 (pruning 50% channel) for the regular resource-constrained moment, 1 (Fire)+ 4 (pruning 1 layer) for the tight memory constraint moment, and 2 (SVD-based decomposition) + 3 (pruning 65% channel) for the tight battery-bounded moment. The evolved models can achieve ≥ 95.6% accuracy and 168.2 ∼ 202.6 arithmetic intensity. AdaSpring searches the proper combinations of compression operators that satisfy diverse demands on accuracy and resource efficiency within 2.8 ∼ 3.1 .

CONCLUSION
This paper addressed the runtime adaptive DNN compression problem to consider the dynamic deployment context of continuously running mobile applications. We present AdaSpring, a context-adaptive and runtime-evolutionary DNN compression framework that continually optimizes the DNN configurations (i.e., architectures and weights) to adapt to the dynamic context. We formulate the dynamic performance demands (e.g. accuracy, latency, energy efficiency) as a time-varying constrained optimization problem. And we propose a heuristic solution as quickly searching for the most suitable combination of retraining-free compression techniques at runtime. To decouple DNN training from runtime adaptive compression, we put computation ahead in the training of a self-evolutionary network at design time (see § 4). And we present the Runtime3C search algorithm and a set of searching speedup mechanisms to boost the runtime search efficiency and quality. Evaluation using five different mobile applications across four mobile platforms and a real-world case study show the performance advantages of AdaSpring to evolve the DNN compression configurations locally online at millisecond level. In the future work, facing the diverse and dynamic mobile scenarios (e.g. data, task, and platform), more efforts and insights for the self-evolutionary deep model compression and optimization frameworks are much needed.