{'fraction_non_alphanumeric': 0.028937808550936132, 'fraction_numerical': 0.007917533455459994, 'mean_word_length': 4.960029607698002, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Decentralized Framework with Dynamic and Event-Driven Container Orchestration at the Edge

Virtualization provides an abstraction layer for the Internet of Things technology to tackle the heterogeneity of the edge networks. It enables the deployment of an application on devices with different architectures to achieve uniformity. This study lays down the fundamentals of a framework for dynamic and event-driven orchestration towards a fully decentralized edge. It provides a blockchain-based delivery platform for containerized applications registered with their resource requirements through a registry on a distributed file system, namely InterPlanetary File System (IPFS). The decentralized resource manager running on the metrics scraped from the host and the virtualization platform, i.e., Docker in our implementation, dynamically optimizes the resources allocated to each container. The framework ensures that variable workloads of a heterogeneous environment can co-exist on multiple edge devices. An event-driven architecture is built over a lightweight messaging protocol, MQTT, capitalizing on the asynchronous and distributed nature of the publish/subscribe pattern to achieve a truly distributed system.


I. INTRODUCTION
The term "Internet of Things (IoT)" describes a sophisticated system of heterogeneous devices, dynamic environments, and complex sub-systems [1]. The quality of IoT services delivered to the end-users is a major concern of the application developers. Edge computing has been proposed to improve user experience by moving services, processing, and actionable insights closer to the user. However, edge devices are also heterogeneous in terms of processing power, memory amount, and operating system. Hence, virtualization schemes, such as containerization, widely used in computer systems have recently stepped into the domain of edge computing [2]. As containers can be scaled according to the available resources of the host system, application developers can present an IoT solution in a container that should ideally execute on almost every edge device [3].
Even though the usage of containers is a breath-taking solution for IoT application developers, it bears its own problems that have to be resolved for a seamless user experience. Firstly, as edge devices are implemented with different types of hardware, container scaling should be done at the edge autonomously. Since each application can have a unique workload during runtime, dynamic load balancing has to be taken into account to maximize resource utilization while scaling the containers. Secondly, the authenticity of the container should be ensured prior to its deployment or upgrade, because malicious containers can overwhelm the host edge device which may be running on scarce resources. Finally, irresponsive or latent services at the edge should be avoided to improve user experience through autonomous deployment of containers to neighboring devices.
In this study, we propose a decentralized framework for autonomous deployment and scaling of containerized applications on resource-constrained edge devices. The secure delivery of containerized applications is enabled by a smart contract that holds container registries on the blockchain. The container images are stored in a decentralized file system, namely, IPFS [4]. The use of blockchain and IPFS provides the ubiquity of the applications for a seamless user experience. Once the user registers to a service or employs an IoT device, the container deployment is done autonomously based on available on-device resources. An event-driven decentralized resource manager is designed for this purpose. It analyzes running services and forecasts future requirements. Based on its findings, it scales containers running on the device. The decision about the new container depends on where the edge device is deployed. If there exist multiple devices at the edge, their resource managers talk, and the device with maximum abundant resources employs the container. Otherwise, only ondevice resources are used in giving the decision. Our study is unique in the sense that it provides an end-to-end solution between the application release and the user experience.
The rest of the paper is organized as follows: The next section presents related studies in the literature. Section III introduces the proposed framework while explaining design choices and incorporated technologies. Section IV analyzes the behavior of the framework under different experimental setups. The final section concludes the work.

II. RELATED WORKS
In the recent literature, there exist several studies on edge container orchestration frameworks. An overview is presented in Table I.
As mentioned in the previous section, the heterogeneous nature of IoT end devices requires scaling. Hence, autonomous scaling (C1) of the containers has to be done at the edge for a seamless user experience. If supported by the system, the options are horizontal (H) or vertical (V). Horizontal scaling refers to changing the number of containers of an application to meet the varying loads. Performance on a single-CPU single-threaded edge device may be degraded in horizontallyscaled containers because having multiple instances of the same application will have to execute sequentially. Thus, [5]- [10] explore horizontal scaling across all available devices in their networks. Vertical scaling is the adaptation of the resources of an existing container. Containerization engines such as Docker provide all necessary tools for vertical scaling, making it inherently more straightforward to configure than horizontal scaling. Scaling type is tightly correlated with software heterogeneity (C3), such as accommodation of different kinds of applications like microservices, batch jobs, and streaming applications on the same device [5], [6], [8], [11]- [14]. Since our target is the utilization of software heterogeneity on resource-constrained edge devices, vertical scaling is preferred.
Decision-taking for autonomous scaling can be either reactive or predictive. Reactive methods only consider the current state. Predictive methods (C2) consider the past and future states of the system. In [10] predictive methods such as machine learning are proposed for horizontal scaling. When vertical scaling is considered, predictive solutions provide a smoother user experience and better decision performance than reactive ones. Our framework relies on statistical models for time-series forecasting as a reliable method that doesn't depend on large amounts of training data. Auto-scaling decisions are based on optimization metrics (C6). The two main categories are application (A) and system (S) metrics. Application metrics are driven by the application requirements such as response time and error rate [7]- [10]. System metrics, such as CPU and memory, are harnessed from the host. Application metrics fail to offer a viable solution when it has to execute on different types of hardware. Hence, our framework utilizes system metrics so that the applications can run on all types of hardware. Frameworks of [3], [5], [6], [10], [12], [15] focus on the resource-constrained devices (C4) as we do, while the rest are deployed on more powerful devices on the edge. This makes on-device orchestration (C5) a challenging issue since it also consumes system resources [3], [6], [7], [9], [10], [13], [15]. In this study, on-device orchestration is also adopted as it supports decentralization.
Edge computing paradigm inherently supports the decentralization of every IoT service [13]. However, the studies in the literature suffer from centralized application registries throughout the whole delivery process. A centralized registry gives the authority to set download rates, storage limits, or pricing decisions to a single party, granting them unfair advantages as well as introducing a single point of failure. Traditional decentralized registry solutions may suffer from some security issues such as access control, immutability, and authenticity of applications. To cope with this limitation, our decentralized delivery process leverages blockchain technol- ogy and a tamper-proof distributed file system to provide proof of ownership on the smart contracts (C8). Decentralized communication between edge devices is also essential. Eventdriven communication (C7) using the publish/subscribe pattern is adopted in many studies [3], [5], [6], [9], [14] as its resilience and robustness make it a popular choice on the edge for building scalable platforms. It is also strategic for decentralized load balancing (C9) across a cluster of edge devices autonomously [7], [13], [15] to distribute the load on other edge devices. Since these works assume trustless setups, a consensus mechanism is implemented. Since we consider secured resource-constrained devices and blockchain is used for secure application delivery, a two-step broadcast followed by an approval-by-silence mechanism is proposed in this work.

III. FRAMEWORK ARCHITECTURE
The proposed framework manages the lifecycle of containerized applications, including design, delivery, execution, and optimization steps. It provides data storage, resource management, monitoring, application registry and delivery solutions, including service upgrades and software updates. The overview of the framework's architecture on the edge is illustrated in Figure 1. The framework on the edge is designed as a distributed set of components to support decentralized orchestration. It consists of four components deployed on the edge device. Monitor scrapes metrics from the host system and shares them with the rest of the framework. Deployer responds to application deployment and resource optimization requests. New deployment requests are accepted through a REST endpoint. Analyzer evaluates incoming requests with the data made available by the other components and makes predictive analysis. Forecaster provides time-series forecasting capabilities to the framework.
The four components of the framework rely on an array of tools and technologies to form the full framework as shown in Figure 1. Docker provides an engine with virtualization   [16]. Blockchain hosts smart contracts that connect the application delivery process with the edge framework.
Host File System is a mounted volume through Docker to grant short-term storage for application metrics on the host device. The framework's components communicate with the publish/subscribe pattern. Message Queueing Telemetry Transport (MQTT) Broker is a prominent lightweight protocol for event-driven architectures on the edge with low resource and power consumption [17]. Here, components publish messages on specific topics to pass events and data for subscribed components to pick up. Messages contains the action field for identification of a message's purpose. Subscribed components act upon the received action and payload of a message. Table  II presents a list of supported actions and their target topics.

A. Application Delivery
The application delivery process provides a pipeline for developers to publish and deliver applications to their clients. The pipeline shown in Figure 2 is built over a decentralized application registry. Versioning, updates, and upgrades are managed on top of IPFS using IPDR as a decentralized and agile solution. IPDR offers the advantage of downloading layers of a Docker image individually with the use of less bandwidth and local storage. In this work, an interface for this decentralized Docker registry is developed as a decentralized application with a smart contract using the data structures and functions in Listing 1. Developers start the Release process of a container application. The application release consists of either Publish, or subsequent Update processes recurring for each service upgrade and software update. First, the released or updated Docker image must be stored on IPDR, which returns an IPFS hash. Then, the application metadata stored on the blockchain is manipulated through the smart contract. The metadata consists of the image hash acquired from IPFS and an optional set of resource limits such as requestedLimit and baseLimit that can be configured by the developer. Data stored on IPFS cannot be traced to its owner unless proof of ownership is deliberately included; nevertheless, the smart contract allows developers to prove the application's authenticity. The delivery process ensures that applications are tamper-proof and only the owner can deliver updates and upgrades.

B. Application Orchestration
The framework components shown in Figure 1 asynchronously pass messages between each other to provide dynamic orchestration shown in Figure 2. The messaging architecture is inspired by the service providers and session messages described in [18]. The orchestration flow is a stepby-step process executed on the edge. It is responsible for managing incoming application deployments and dynamically adapting allocated resources for each container. The selfadaptive properties are modeled after Monitor-Analyze-Plan-Execute-Knowledge (MAPE-K) loops [19].
1) Deployment Admission: The deployment request is initiated by a request that is made externally using the REST API provided by the Deployer. The actor invoking the request can either be the user or an IoT device connected to the network. Then, the Deployer acquires the image hash and resource limits stored on the blockchain via the smart contract. After pulling the corresponding application image from the registry, a deployment analysis request is published with the resource limits on the analysis topic.
2) Resource Allocation: Each container is deployed with a set of CPU and memory limits specified in the smart contract. These limits are used in scaling the containers based on the status of other active deployments and vendor-defined resource limits. The Analyzer is subscribed to two different topics: analyze and forecast. It listens to deployment analysis requests and publishes a corresponding deployment analysis response on the analyze topic as shown in Table II. The analysis determines the feasibility of a new deployment with its resource requirements.
The analysis takes the future availability of the resources into account rather than their current values to ensure the long-lasting stability of the system. Therefore, a time-series forecasting analysis is conducted by the Forecaster. It is subscribed to the forecast topic and acts on a forecast request messages published by the Analyzer. The Autoregressive Integrated Moving Average (ARIMA) model is used to interpret the trends in time-series data and predict the future values with statistical analysis [20], [21]. The time-series data consist of the metrics collected and stored locally by the Monitor as described in Section III-B4. Auto-Regression and Integrated models of ARIMA are configured together to capture nonstationary patterns seen in each analyzed metric and forecast their values. The ARIMA model used in our implementation is with an order of (5,1,0), which can capture the short time trends in available metrics data. This configuration sets up a lag order of 5 to smooth the time-series data and a degree of differencing of 1. The use case scenarios depend on the environment where the edge unit is deployed. This work focuses on daily patterns which can be observable in smart homes or offices. In order to take this temporal characteristic into account, data points are aggregated hourly.
The prediction results for each container are its predicted CPU and memory utilization based on its lifecycle. These results, P util r,c , are published on the forecast topics so that Analyzer will be able to compute the feasibility of deploying a new container. To achieve this, it has to compute system availability, P avail r,S . Existing containers' resources are always prioritized over a new deployment. Hence, their resources, L current r,c , are not lowered. Therefore, P util r,c is updated as P ) for each C c and R r . The symbol list is given in Table III. The analysis request includes L target r,c which can be one of the resource limit values: L request r,c , or L base r,c . In their absence, a default value is used. The Analyzer approves a deployment request if L target r,c ≤ P avail r,S , meaning that the system will have enough resources to accommodate the incoming application deployment. Otherwise, the request is rejected.
3) Container Execution: Deployment analysis responses received by the Deployer can be twofold, corresponding to either a deployment accept or a deployment cancel as shown in Table II   . Optimization for memory limit sets L target mem,c , which is bounded by system-defined minimum and maximum resource values, L min mem and L max mem . The upper limit ensures that a single container cannot allocate a very high share of system memory. The lower limit protects the container from scaling down indefinitely. The amount for scaling up and down, L scale mem,up and L scale mem,down , are constants that can be configured in the framework. However, since C util mem,c exceeding L current mem,c is a risk that can kill the container, a margin is defined between P util mem,c and L target mem,c while scaling up. If P util mem,c is less than C util mem,c , the Analyzer computes L target mem,c as L current mem,c − L scale mem,down to scale down the container. After the L target mem,c is set, in case of downscaling, the new limit is once again compared with max(P util mem,c ) to make sure that the container is not scaled below the allocated margin. Otherwise, L current mem,c is returned, and memory is not scaled down any further. For scaling up, a similar approach is adopted.
Optimization for CPU limit sets L target cpu,c . If max(P util cpu,c ) is greater than L current cpu,c , container is upscaled with L target cpu,c := L current cpu,c + L scale cpu,up . In the opposite case, container is downscaled with L target cpu,c := L current cpu,c − L scale cpu,down . Setting a limit on the CPU causes throttling in Docker [22]. As high amount of C throttle c severely degrades the application performance, two extra precautions are taken while downscaling the containers that have already been throttling. Firstly, P throttle c is compared against L throttle which is a constant defined in our framework. If max t (P throttle c [t]) is greater than L throttle , an adjustedScale amount where adjustedScale := (L scale cpu,up × max t (P throttle c [t]))/100 is used instead of L scale cpu,up for a minor scale up. The second control is applied before returning L target cpu,c , which can cause throttling after a scale-down. In order to prevent sudden throttling, L target cpu,c is recalculated as L target cpu,c := max(P util cpu,c ) × L buf f er cpu , which is set to 110% in our framework. Buffer ensures the application to have enough room for unexpected load spikes until the next optimization without causing unnecessary throttling. After this adjustment, it is possible that L target cpu,c can be above L current cpu,c . In that case, any scaling operations are overturned by returning L current r,c as the throttling was already below L throttle .

C. Cluster Deployment
The proposed framework can be deployed on multiple edge devices as a cluster in the same network. This strategy supports not only load balancing across the connected hosts but also decentralized orchestration of deployments. The setup depends on MQTT bridges where brokers can automatically broadcast configured messages in the network. Each device's IP address is provided to each other broker to set up the bridges. However, it is also possible to set up an MQTT device discovery mechanism to automatically find other brokers on the network [23]. Shared topics, monitor and deploy, are configured and prefixed with the cluster keyword so that each component can differentiate internally or externally generated messages.
Deployer on each device is subscribed to both monitor and cluster/monitor topics. Key/value pairs of IP addresses and S avail r are generated by the Deployer for each device and updated with each monitoring result message. Deployer uses these pairs to track the available resources of all devices in the network. It is also subscribed to the deploy and cluster/deploy topics where it receives deployment requests made on all devices. When a new request is received, each Deployer selects the device with the highest amount of available resources from its key/value pairs. Then, only the device that decides itself as the best candidate continues with the deployment by publishing a deployment analysis request. Cluster deployment follows the same steps as the regular deployment workflow. If S avail r of multiple devices are the same, each device marks the device with the smallest IP address for deployment.

IV. EXPERIMENTS AND RESULTS
All experiments are carried out on Raspberry Pi 4 Model B with a 64-bit quad-core Cortex-A72 processing unit, 8GB LPDDR4-3200 SDRAM, and Broadcom BCM2711. A slice with 1 CPU and 1GB memory with disabled swap usage is created to emulate a resource-constrained device. Docker daemon's cgroup parent is assigned to this slice to limit the maximum allowed resource usage. CPU limits and utilization are given in CPU units. 1 CPU is equal to a single CPU core, and 50% of this core is represented as 500mCPU. The framework also exists on the same hardware. The CPU utilization is negligible for most components except for the Forecaster and IPFS. The Forecaster utilizes all available CPUs during forecasting to produce results as quickly as possible. IPFS utilizes 50-250 mCPU on average. It can be reduced by switching to a minimal setup, disabling unnecessary features, and limiting the peers. The memory footprint of the entire framework is currently 400MB, which can be reduced significantly by pruning and optimizing the libraries.
In order to create a controlled environment for the tests, containerized applications are prepared for each resource type. Five workload patterns proposed in [24] for container-based cloud applications are adapted to this work to generate workload patterns that represent different IoT application types: 1) Slowly rising/falling workload pattern 2) Drastically changing workload pattern 3) On-off workload pattern 4) Gently shaking workload pattern 5) Real-world workload pattern The IoT applications can be classified as data-dominant and CPU-dominant. Hence, ten containerized applications are generated. Labels for each workload pattern are referenced with the class name. For example, a slowly rising/falling pattern for the data-dominant workload is identified as Memory 1. The same pattern for CPU-dominant workload is represented by CPU 1. The experiments are carried out on each group separately. This approach does not exclude the case where a CPU-dominant application coexists with a data-dominant application on the same device as different algorithms execute for memory optimization and CPU optimization. The maximum memory amounts in these workloads are 95MB, 95MB, 95MB, 80MB, and 95MB, respectively, as listed above. Similarly, 150m, 150m, 150m, 120m, and 140m represent the maximum CPU utilization. The experiments are carried out on each group separately. This approach does not exclude the case where a CPU-dominant application coexists with a data-dominant application on the same device as different algorithms execute for memory optimization and CPU optimization. The experimental results are presented in Figure 3 where solid and dashed lines show workloads and resource limits, respectively.

A. Orchestration of Multiple Containers on a Single Device
These experiments are carried on for a single device that has to host multiple containers with different limit settings and deployment scenarios. The first two scenarios consider application deployments in an extremely short period. This usually happens when the device is installed for the first time. The third scenario simulate deploying new applications on an already running device. The fourth scenario handles the case where the host is extremely low on resources 1) Sequential Deployment Requests with Ample Resource Limits: In this scenario, the application limits specified are above the expected peak memory usage of the workloads. For each data-dominant workload, L request mem,c = 150M B and L base mem,c = 100M B. As shown in Figure 3(a.1-5), all five workloads start simultaneously. The memory limits converge to the actual usage in a few iterations of optimization steps every 5 minutes. The initial delay before the first optimization allows the framework to collect enough metrics to make utilization predictions for the upcoming optimization interval.
In is accepted, the deployment is rejected because of the limited resource of the system. Note that total request makes 1500mcore which is more than the allocated CPU on the emulation hardware. Thus, the first deployment of the fourth workload with L request cpu,c is rejected and deployed with L base cpu,c in a second attempt by the framework, as shown in Figure  3(b.1-4). With the first four workloads running on the system, neither L request cpu,c nor L base cpu,c for the fifth workload can be satisfied (Figure 3(b.5)). The limits of the successfully deployed workloads are initially lowered drastically with a constant L scale cpu,down , which is later adjusted based on P util cpu,c and P throttle c resulting in more minor decrements.

2) Sequential Deployment Requests with Drastically Low
Resource Specifications: This scenario assumes that all limits are given much lower than the actual requirements. In this experiment, we set L request mem,c = 15M B and L base mem,c = 10M B . All five memory workloads fail to start for the first couple of tries, as shown in Figure 3(c1-5). After the initial two deployment attempts with no prior knowledge of the application, the framework begins assigning by rapidly increasing L target mem,c based on L base mem,c incremented by L scale mem,c by several retry counts. Then, after the first four tries, Memory 1 and Memory 2 can be deployed as their resource usage is comparatively lower than the rest in the first one-third of their period. Other workloads keep getting killed as they immediately try to allocate more memory than allowed by L current mem,c . These burst restarts cause their L target mem,c to increase so rapidly that this limit surpasses C util mem,c . However, once the containers are deployed, their limits are lowered by L scale down,mem until a tighter fit based on P util mem,S is found. The deployment behavior of the CPU workloads vastly differs from the memory workloads. This is due to the absence of an error similar to out of memory. after a couple of optimization cycles, eventually matching the limits similar to those in Figure 3(b. [1][2][3][4][5]. Note that the delay is removed. The deployment behavior of the CPU workloads vastly differs from the memory workloads, as shown in Figure 3(c. [1][2][3][4][5]. This is tied to the absence of an error similar to out of memory. Therefore, all five deployments are deployed with their misconfigured L request cpu,c values. These workloads typically exhibit a C util r,c higher than this L current cpu,c . Consequently, they are all immediately throttled. Fully throttled containers exhibit delays in their workload which is demonstrated in Figure  3(d.1-5) for workloads CPU 1-5. The framework optimizes L current cpu,c responsible for the throttling after a couple of optimization cycles, eventually matching the limits similar to those in Figure 3(b.1-5).
3) Interleaved Deployment of Workload Pairs: This experiment studies the effects of introducing a new container to a stabilized system. For this purpose, after the first two workloads are optimized, deployment requests from the last two workloads will be received.
As shown in Figure 3(e. [1][2][3][4] and Figure 3(f.1-4), the outcome is the same for CPU and memory workloads. The optimization steps adapt to the new workloads without interfering with the past deployments. The optimization of the newly introduced applications starts once enough data is collected for predictions to be performed.

4) Extreme
Resource Constraints on the System: Some hosts can be running on extremely low resources. Pre-existing deployments can impose very tight constraints on the new requests. To simulate this environment for data-dominated applications, we set S avail mem = 400M B and studied with the first three memory workloads. The first two workloads are deployed with their L request mem,c (Figure 3(g.1-2)). The third workload cannot be deployed as there is not enough memory for the third workload's L request mem,c or L base mem,c . Another experiment is run with a similar configuration with S avail mem = 200M B. As shown in Figure 3(h.1), only the first deployment is accepted as any subsequent deployments would require more memory than S total mem . Similar results are obtained from the same experiments for CPU images. We set S avail cpu = 350m and deployment requests for the first three workloads are sent. Only the first one is accepted with L request cpu,c as shown in Figure 3(i.1). However, S avail cpu is not enough to deploy the remaining two workloads. These workloads are tried twice with L request cpu,c and L base cpu,c before they are rejected. Then, the experiment is rerun with even tighter resources where S avail cpu = 100m. A similar deployment pattern can be seen in in Figure 3(j.1). The only difference is that the first workload is deployed with L base cpu,c . This container exhibits delays at regular intervals; nevertheless, it cannot be scaled up due to lack of S avail cpu .

B. Experiments with Cluster Deployment
This experiment is devised to observe the load balancing capabilities of the framework across multiple devices. Three edge devices with identical resources, S avail r , are used for this purpose. Device IP addresses increase from device 1 to 3, where device 1 is the device with the smallest IP address. The first device receives sequential deployment requests from the first four memory workload patterns, i.e. Memory 1-4.
Firstly the MQTT broker on each device starts, and the framework is deployed. Then device 1 receives Memory 1 workload. It deploys it on itself since its IP is the smallest. Following this deployment, all three devices synchronize their resource availability key/value pairs with monitoring result messages published by device 1. With the second request, each device identifies device 2 as the processor. The same steps are followed for the remaining two applications. The third application is deployed on device 3 which evens up the resource availability of all three devices. The final application is deployed on device 1 which has the smallest IP address. It should be noted that the cluster deployment does not introduce latency overhead during deployments. Only a single extra event is exchanged over the cluster/deploy topic if another device satisfies the deployment request while the rest of the workflow stays the same.

V. CONCLUSION
In this paper, we have presented an end-to-end solution for a seamless user experience between application developers and end-users on the IoT edge. The exploitation of virtualization and decentralization technologies transformed heterogeneous and resource-constrained edge gateways into vendor-agnostic hosts for IoT applications. Our framework lowers the edge computing footprint by sharing hardware and maximizing resource utilization with autonomous scaling and load balancing in the presence of multiple devices. The event-driven resource manager establishes a fair and reliable platform for all involved parties based on analysis of running services and forecasts of future requirements. Concerns about the authenticity of deliveries such as releases and upgrades are ensured through smart contracts and distributed files system. Specific aspects of the framework can be effortlessly extended due to its modular architecture of message queues and distributed components.