{'fraction_non_alphanumeric': 0.026824278245055694, 'fraction_numerical': 0.007729029324846556, 'mean_word_length': 5.0850622406639, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SplitPlace: Intelligent Placement of Split Neural Nets in Mobile Edge Environments

In recent years, deep learning models have become ubiquitous in industry and academia alike. Modern deep neural networks can solve one of the most complex problems today, but coming with the price of massive compute and storage requirements. This makes deploying such massive neural networks challenging in the mobile edge computing paradigm, where edge nodes are resource-constrained, hence limiting the input analysis power of such frameworks. Semantic and layer-wise splitting of neural networks for distributed processing show some hope in this direction. However, there are no intelligent algorithms that place such modular splits to edge nodes for optimal performance. This work proposes a novel placement policy, SplitPlace, for the placement of such neural network split fragments on mobile edge hosts for efficient and scalable computing.


I. PROBLEM AND MOTIVATION
Modern Deep Neural Networks (DNN) are becoming the backbone of many industrial tasks and activities [1]. As the computational capabilities of devices have improved, new deep learning models have been proposed [2]. Such neural models are becoming increasingly demanding in terms of data and compute power to provide higher accuracy results in more challenging problems. Many recent DNN models have been shown to outperform the earlier shallow networks for more complex tasks like image segmentation, traffic surveillance, and healthcare [3]- [5]. Moreover, recently paradigms like mobile edge computing have emerged which provide robust and low latency deployment of Internet of Things (IoT) applications close to the edge of the network.
However, mobile edge devices face the severe limitation of computational and memory resources as they rely on low power energy sources like batteries, solar or other energy scavenging methods [6], [7]. This is not only because of the requirement of low cost but also the need for mobility in such nodes [8]. Herein, it is still possible to handle the processing limitations of massive DNN models by effective preemption and longer execution of jobs. However, memory bottlenecks are much harder to solve [8]. In a distributed edge environment where storage spaces a typically mapped to a networks-attached-media, large swap spaces impose very high network bandwidth overheads making high fidelity inference using DNNs hard [4], [9]. To deploy an upgraded AI model, tech-giants like Amazon, Netflix and Google need to revamp their infrastructure and upgrade their devices, raising many sustainability concerns [1]. This has made the integration of massive neural network models with such devices a challenging and expensive ordeal.
The only solution for this problem is the development of strategies that can accommodate large-scale DNNs within legacy infrastructures. Many prior efforts in this regard have been proposed [10]- [12] but they fail to provide a holistic strategy for not only distributed learning but also inference in such memory-constrained environments. This abstract proposes to solve these challenges by intelligent splitting and placement of large neural models into modular fragments. This project is part of a larger endeavor to efficiently integrate the hitherto disjoint fields of advanced deep learning and distributed systems research.

II. BACKGROUND AND RELATED WORK
Recently, many research ideas have been proposed like Cloud AI, Edge-AI and Federated learning which aim to solve the problem of running enormous deep learning models on constrained edge devices [11], [12]. However, Cloud-AI faces the problem of high average response time making it unsuitable for latency-critical applications like Healthcare [4], [13]. Edge-AI requires all training data to be centralized leading to high bandwidth overheads [11]. Federated learning depends on data distribution over multiple nodes and assumes that neural models with data batches can be accommodated in the system memory. This is seldom the case for low-end nodes like Raspberry Pis [14].
Other recent works propose lower precision models that can fit in the limited memory of such devices by using methods like Model Compression [15], [16]. However, compressed and low-precision models lose the inference accuracy making them unsuitable for accuracy sensitive applications like security and intrusion detection [17]. Recently, split neural network models have been proposed which show that using semantic or layer-wise split, a large deep neural network can be split to multiple smaller networks for dividing network parameters into multiple nodes [10], [18]- [20]. However, no appropriate scheduling policies exist which can intelligently place such modular neural fragments on a distributed infrastructure to optimize both accuracy and Service Level Agreements (SLA).

A. Layer and Semantic Splits
Semantic splitting divides the network weights into a hierarchy of multiple groups that use a different set of features which  produces a tree structured model which has no connection among branches of the tree allowing parallelization of input analysis. Due to limited information sharing among neural network fragments, the semantic splitting scheme gives lower accuracy in general. Semantic splitting requires a separate training procedure where pre-trained models can not be used, however, it provides parallel task processing and hence lower inference times more suitable for mission-critical tasks like healthcare and surveillance. Layer wise splitting divides the network into groups of layers for sequential processing of the task input. Layer splitting is easier to deploy as pretrained models can be just divided into multiple layer groups and distributed to different mobile edge nodes. However, layer splits require semi-processed input to be forwarded to subsequent edge node with final processed output to be sent to the user, thus increasing the overall execution times. Moreover, layer-wise splitting gives higher accuracy compared to semantic splitting. An overview of these two strategies is shown in Figure 1.

B. Intelligent Placement
However, the placement of such split models is non-trivial considering the diverse and complex dynamism of task distribution, model usage frequencies and geographical placement of mobile edge devices [21]. This work proposes a novel split placement policy, SplitPlace, for enhanced computation at the edge of the leveraging mobile edge platform to reach low latency results as well as allowing modular neural models to be integrated for best result accuracies that only cloud deployments could provide till date. The idea behind the proposed placement policy is to maintain moving average estimates of the time it takes for the complete execution of the "layer" split decision (E a for application a ∈ A). Based on the SLA deadline of a new workload, we run two Multi-Armed-Bandit (MAB) models to estimate the "expected reward" of each decision. Motivation being that SLA deadline being lower than the execution time of a layer split, such a decision would more likely lead to a SLA violation. The reward for a sequence of decisions on a set of workloads (W ) is expressed as where SLA w is the deadline of workload w. This method is agnostic to the underlying scheduling strategy. However,  for comparison with baselines, we combine it with a popular Asynchronous-Actor-Critic scheduler [8].
These MAB models are learned and combined with an intelligent decision-aware scheduler for the dynamic placement of DNN workloads on a mobile edge computing environment. An overview of the proposed method is shown in Figure 2 where D t is the decision (layer or semantic splitting) for workload w t received at time t with application type a ∈ A. Finegrained decision making on how to optimally split DNNs can be explored in the future.

IV. PERFORMANCE EVALUATION
To compare the proposed approach, we use a resourceconstrained edge setup with 10 Raspberry-Pi like devices having 4 to 8 GB of RAM available per device. To emulate mobility, we use Gaussian noise in the network latency using the netlimiter Linux tool [8]. We use a popular imageclassification models of ResNet50-V2, MobileNetV2 and In-ceptionV3 [16]. We use popular model-compression strategy as a baseline giving low memory-footprint versions of this model to run on the test setup as vanilla models can not be run directly in such constrained devices [16], [22]. Table I shows the results of our experiments. With a 10.6% higher scheduling time compared to baseline, the SplitPlace model is able to give lower energy consumption and SLA violation by 5% and 61% respectively. Moreover, the proposed model gives nearly 3% higher inference accuracy and 9.6% higher average reward compared to the baseline.