{'fraction_non_alphanumeric': 0.02687558784094775, 'fraction_numerical': 0.009045246242820836, 'mean_word_length': 5.302434102661118, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Harvesting Idle Resources in Serverless Computing via Reinforcement Learning

Serverless computing has become a new cloud computing paradigm that promises to deliver high cost-efficiency and simplified cloud deployment with automated resource scaling at a fine granularity. Users decouple a cloud application into chained functions and preset each serverless function’s memory and CPU demands at megabyte-level and core-level, respectively. Serverless platforms then automatically scale the number of functions to accommodate the workloads. However, the complexities of chained functions make it nontrivial to accurately determine the resource demands of each function for users, leading to either resource over-provision or under-provision for individual functions. This paper presents FaaSRM , a new resource manager (RM) for serverless platforms that maximizes resource efficiency by dynamically harvesting idle resources from functions over-supplied to functions under-supplied. FaaSRM monitors each function’s resource utilization in real-time, detects over-provisioning and under-provisioning, and applies deep reinforcement learning to harvest idle resources safely using a safeguard mechanism and accelerate functions efficiently. We have implemented and deployed a FaaSRM prototype in a 13-node Apache OpenWhisk cluster. Experimental results on the OpenWhisk cluster show that FaaSRM reduces the execution time of 98% of function invocations by 35.81% compared to the baseline RMs by harvesting idle resources from 38.8% of the invocations and accelerating 39.2% of the invocations.


Introduction
The emergence of serverless computing has extensively simplified the way that developers access cloud resources. Existing serverless computing platforms, such as AWS Lambda, Google Cloud Functions, and Azure Functions, have enabled a wide spectrum of cloud applications, including video processing [3,9], data analytics [17,23], and machine learning [6,14] with automated resource provisioning and management. By decoupling traditional monolithic cloud applications into inter-linked microservices executed by stateless functions, serverless computing frees developers from infrastructure management and administration with finegrained resource provisioning, auto-scaling, and pay-as-yougo billing [18].
The serverless architecture has also introduced new challenges in computation efficiency, resource management, and task scheduling for cloud providers, attracting a rich literature of research studies. Some focused on optimizing the low-level virtualization layer to mitigate function cold-starts and enhance resource isolation [1,2,8,11,24]. Some proposed new resource management and function scheduling algorithms to accelerate serverless applications and improve resource utilization [12,[30][31][32].
However, existing serverless services are still suffering from low resource efficiency due to the inappropriate resource allocations requested by users, who are uncertain about the exact function resource demands. Decoupling monolithic cloud applications to a serverless architecture generates numerous types of functions and complex interfunction dependencies [12,29]. Though serverless functions have much fewer configuration knobs than the traditional cloud virtual machines, it is still tricky for most users to estimate the accurate memory and CPU demands for each function. Existing serverless computing platforms apply static strategies to provision resources for functions. For example, AWS Lambda allocates CPU resources to a function proportionally to its memory size configured by users [5], leading to either CPU over-provision or under-provision for the function execution. Therefore, serverless service providers are enduring poor resource utilization due to users' inappropriate function configuration-some functions are assigned with more resources than they need [12]. The high concurrency and fine-grained resource isolation of serverless computing further amplify such inefficient resource provisioning.
A few studies have attempted to address the above issues. Shahrad et al. [30] and FaasCache [10] proposed to maximize resource utilization and reduce the number of cold-starts by predicting the keep-alive windows of individual serverless functions. Fifer [12] incorporated the awareness of function dependencies into the design of a new resource manager to improve resource utilization. Besides, several research works aimed to accelerate functions and improve resource efficiency by adjusting CPU core allocations for serverless functions in reaction to their performance degradation during function executions [19,20,32]. Though, users still have to configure memory size for each function.
However, none of the existing studies has directly tackled the low resource efficiency issue raised by the inappropriate function configurations. There are three critical challenges in the perspective of serverless service providers to address this issue: First, a user function is secured as a black box that shares no information about its internal code and workloads, making it hardly possible for the serverless system to estimate user functions' precise resource demands. Second, decoupling monolithic cloud applications to serverless computing architectures generates a variety of functions with diverse resource demands and dynamic input workloads. Third, the resource provisioning for serverless functions is fine-grained spatially (i.e., small resource volumes) and temporally (i.e., short available time).
We present FaaSRM, a new serverless resource manager that dynamically harvests idle resources to accelerate functions and maximize resource utilization. The intuition behind FaaSRM is to carefully harvest idle computing resources from functions over-supplied and accelerate functions undersupplied without degrading the performance of functions with resources harvested. FaaSRM monitors a series of performance metrics and resource footprints, including CPU utilization, memory utilization, and function execution times to estimate the actual resource demands of running functions. We apply an experience-driven algorithm to identify functions over-supplied and under-supplied. FaaSRM learns to accurately determine the volume and available time of idle resources for harvesting from over-supplied functions to under-supplied functions. By decoupling the fixed proportion between CPU and memory resource allocation, FaaSRM adjusts both types of resources independently. We also design a safeguard mechanism to guarantee that FaaSRM's resource harvesting leads to no performance degradation. To deal with the highly volatile environment of serverless computing and large numbers of concurrent functions, we propose to apply the Proximal Policy Optimization (PPO) algorithm [28] to learn from the realistic serverless system and make per-invocation resource adjustments.
We implement FaaSRM based on PyTorch with multiprocess support in the Apache OpenWhisk platform [4]. We evaluate FaaSRM with other three baseline RMs in simulation and OpenWhisk experiments using Azure Functions traces [30]. Compared to the default RM in OpenWhisk, FaaSRM reduces the execution time of 98% function invocations 1 by 87.60% and 35.81%, in simulation and Open-Whisk, respectively. Particularly, FaaSRM harvests idle resources from 38.78% of function invocations while accelerating 39.18% on the OpenWhisk cluster. Besides, FaaSRM only degrades the performance of a negligible number of function invocations under the system performance variations of the OpenWhisk cluster.

Background and Motivation
This section will first briefly introduce serverless computing systems and resource provisioning strategies. We then use a realistic example to motivate the necessity to improve 1 In this paper, a function refers to a runnable code deployed on serverless platforms, an invocation refers to an instance of executing the function. resource utilization of serverless computing platforms. We also show how to optimize serverless computing resource provisioning with reinforcement learning.

Resource Provisioning in Serverless Computing
Generally, serverless computing users are only responsible for uploading source codes and choosing resource limits for their applications. Serverless platforms provision resources only when users invoke their functions. Requesting too much resource leads to low resource utilization and extra bills for users. Thus insufficient resource provision leads to poor performance. However, it is non-trivial for users to figure out a performant and cost-efficient resource configuration for their applications. To make the right decision, users must fully understand both functions and platforms, which may require countless testing and profiling before the actual deployment.
Typical serverless applications such as image processing, machine learning inference, data analytics are stateless and event-driven. The performance is mainly dominated by computing resources, such as CPU and memory. Existing FaaS platforms, such as Apache OpenWhisk and AWS Lambda, request users to define the memory limit for their functions and then allocate CPU limit proportional to the memory. For example, Wang et al. [33] identify that both AWS Lambda and Google Cloud Function allow users to utilize CPU proportionally to function memory. Such resource management policies could lead to various problems. For example, it encourages users to over-provision memory to accelerate their functions, leading to memory under-utilization. To mitigate this issue, ENSURE [32] proposes to decouple CPU allocation from memory and automate CPU adjustment based on performance degradation detection.
However, such decoupling is hard since functions in serverless platforms have bursty invocations and repeatable workloads. For example, the analysis of characteristics of serverless applications running on Microsoft Azure Functions [30] indicate that most serverless workloads have apparent patterns in the traces of duration, invocation, and resource consumption. To handle such dynamics, we propose a novel resource management framework for serverless platforms, FaaSRM, which fully manages resources from users. FaaSRM manages and provisions computing resources by learning the patterns behind serverless platforms and functions. As far as we know, no existing FaaS platform can intelligently manage and provision resources for serverless functions based on their workload patterns.

A Motivating Example
To illustrate our motivation, we first conduct a simple experiment using the following two representative resource managers (RMs) to show the performance improvement when adopting an RM that dynamically allocates resources for serverless functions: Fixed RM is the default RM employed by most existing serverless platforms, requires users to pre-define memory for their functions, and then consistently provisions the exact amount of memory. CPU resources are then allocated proportionally to functions based on the memory. We assume users configure their memory and make no changes after invocation starts. Greedy RM proactively optimizes resource allocation based on fixed step size. For each function invocation, it allocates resources based on function's recent resource usage. We simulate a 60-second workload with online invocation arrivals on our simulator. We randomly sample six functions from real-world serverless function traces released by Microsoft Azure Functions [30]. For Fixed RM, we preconfigure the CPU and memory of six functions according to memory limits from the corresponding memory traces. For Greedy RM, we initially allocate the same amount of resources with Fixed RM. We configure the simulating cluster with ten servers, and each server has 8 CPU cores and 2 GBs memory available. Each function has access to 8 CPU cores and 512 MBs memory at most. We describe the details of workloads and the simulator in Section 6. Figure 1 shows the average function execution time (FET) of individual functions. In contrast to Fixed RM, Greedy RM allocates resources proactively, resulting in significant improvement on under-provisioned functions. Figure 2 shows the cumulative distribution function (CDF) of FETs of invocations for two RMs provisioning the same workloads. Greedy RM completes most invocations faster than Fixed RM. Figure  3 shows the average CPU allocation of individual functions for Fixed RM and Greedy RM. Compared to Fixed RM, Greedy RM harvests idle CPU cores from over-provisioned functions (function 2, 4, 5, and 6) without significantly deteriorating their performance and accelerates under-provisioned functions (function 1 and 3) with additional resource. Given the same environment and workload, Greedy RM outperforms Fixed RM by significantly improving the entire workload.
However, human-designed resource provisioning policies can hardly comprehend both characteristics of platforms and functions, which can result in certain problems. For example, Greedy RM may fail to handle a load spike due to over-harvesting or offer more resources than needed when supplementing functions. This paper argues for a new resource manager for serverless platforms using deep reinforcement learning, FaaSRM, which dynamically and safely harvests idle resources from over-supplied functions to support under-supplied functions.

Deep Reinforcement learning
Reinforcement learning (RL) is a machine learning technique that an RL agent learns to make sequential decisions by autonomously interacting with the environment. At every timestep , the agent is in a specific state , and evolves to state +1 according to a Markov process with the transition probability P( , , +1 ) when action is taken. The immediate reward for the agent to take action in state is denoted as . The goal of the agent is to find a policy that makes decisions regarding what action to take at each timestep, i.e., ∼ (·| ) so as to maximize the expected cumulative rewards, i.e., E [ ∞ =0 ]. However, it was tricky to apply RL to optimize real-world systems due to the large state space. To address this challenge, deep reinforcement learning (DRL) has been proposed to optimize scheduling and resource provisioning in distributed systems [21,22]. More specifically, deep learning provides function approximators while RL algorithms describe how to train the weights of an arbitrary function approximator for sequential decision problems. As a result, DRL simply takes deep networks as a non-linear function approximator, which outputs computable functions that depend on a set of adjustable parameters, , which we can adjust to affect the behavior of a policy via some optimization algorithms. We refer as policy parameters and represent the policy as ∼ (·| ). In DRL, Deep Neural Networks (DNNs) have been the most popular function approximators to solve stochastic RL tasks, as hand-crafted features are not required by DNNs. We consider resource provisioning for serverless computing systems as a stochastic large-scale RL task, hence we use a neural network to implement our FaaSRM policy.

Objectives
Given serverless workloads, FaaSRM rebalances resources at the function level, i.e., FaaSRM harvests idle resources from over-provisioned functions to improve under-provisioned functions. We focus on the function execution time (FET), which is the time from starting its execution until completion, of serverless functions as FET mostly depends on computing resources (CPU and memory). We focus on function execution time rather than initialization overhead or queuing delay, but optimizations on the two objectives can be complementary to our work. As FaaSRM accelerates underprovisioned function invocations by reducing the FET with supplementary resources, we must ensure that those functions being harvested have no obvious performance degradation. Hence, FaaSRM targets three objectives: harvesting idle resources from over-provisioned functions, accelerating under-provisioned functions, and guaranteeing no function suffering from obvious performance degradation.

FaaSRM's Architecture
We present FaaSRM, a resource manager in serverless platforms that dynamically improves resource allocation for functions. Figure 4 depicts an overview of FaaSRM's architecture. Serving workloads in a serverless platform, FaaSRM offers an optimal resource allocation predicted by an RL agent for each function invocation of workloads. To provide an allocation option for an incoming function invocation, FaaSRM collects information of the platform and the function, and embeds them into a state as input to its RL agent. The RL agent computes an optimal allocation option based on the given information. Then FaaSRM applies the option to the invocation by communicating with the platform.
Serverless functions generally have different demands for CPU and memory. CPU-intensive applications are forced to over-provision memory to acquire more CPU power, which leads to memory under-utilization. To locate the needs of two resources for functions independently, FaaSRM decouples CPU and memory allocation binding in existing serverless platforms such as AWS Lambda and OpenWhisk, and manages CPU and memory using an identical but independent system. FaaSRM allows users to specify both CPU (cores) and memory (MBs) for their functions, and employs the userdefined resources as baselines in the safeguard mechanism to guarantee Service Level Objectives (SLOs) of individual function invocations.
Harvesting idle resources from "over-provisioned" functions can be dangerous, especially with RL predictions. Performance can degrade when FaaSRM under-predicts the resource demands and harvests from functions that can utilize full capacity. To avoid obvious performance degradation of harvested functions, FaaSRM applies a safeguard mechanism to prevent those potentially dangerous allocation options and guarantees the SLOs of every function invocation within a workload. The safeguard works on top of allocation options provided by FaaSRM's RL agent. After deployed in a platform, FaaSRM keeps a history of recent requests for each function. Before selecting an option for the incoming invocation, FaaSRM filters out all allocation options below the peak from amongst the historical allocation decisions. Once the last request usage exceeds allocation (exceeds 90% of the allocation in Algorithm 1), FaaSRM immediately calls a safeguard invocation to return all harvested resources to the function, i.e., allocating the exact amount of resources defined by users, in the next request. FaaSRM then recalibrates the function's baseline by taking the execution time of safeguard invocation. Hence, FaaSRM improves the overall performance of workloads while ensuring SLOs of individual function invocations by regulating its decision-making process.

Challenges
We tackle three key challenges of rebalancing resources for serverless functions: Decouple CPU and memory. Existing serverless platforms bind CPU with memory to improve resource utilization of servers, i.e., allocate CPU power proportionally to the memory. To harvest idle resources independently, it is non-trivial for platforms to decouple CPU and memory from binding them together. Furthermore, decoupling CPU and memory introduce two independent decision dimensions of resource management. To make good allocation decisions, the resource manager has to identify the demands of both CPU and memory of functions.
Huge action space. To provision supplementary resources to under-provisioned functions, FaaSRM has to select an allocation option from a two-dimensional resource pool, leading to an immense action space. For example, AWS Lambda allows users to configure memory from 128 MB to 10240 MB and have access to 6 CPU cores, in total 60,672 choices. A huge action space can result in a neural network with a giant output size, which is inefficient and difficult to train.
Performance degradation. While FaaSRM harvests resources from functions deemed as over-provisioned and improves the entire workload, one necessary requirement is to prevent the performance of those functions from degrading. It is vital to guarantee SLOs of individual functions, i.e., harvested functions have no significant performance degradation.
To address the first two challenges, we propose a RL agent using score function to make decisions inside FaaSRM. By reusing the same score function, we manage to decouple CPU and memory resources, and make two-dimensional decisions with small, fixed-size of action space. For the last challenge, we devise a safeguard mechanism to protect performance of functions harvested by FaaSRM.

FaaSRM's Design
In this section, we present the design of FaaSRM. We first formulate the resource harvesting and provisioning problem in serverless computing and then introduce the information collection and embedding procedure. We describe the score function based policy network in FaaSRM and how it allocates resources for function invocations. We also describe the safeguard mechanism atop FaaSRM for preventing individual functions from significant performance degradation. Finally, we present the training algorithm for FaaSRM.

Problem formulation
A serverless platform has multiple worker nodes. On each node, we focus on the two main resource types in a serverless environment, i.e., CPU and memory. We assume that all nodes have homogeneous hardware configurations and the total available resource of the platform is limited. We consider functions with the same resource demands for CPU and memory. The resource profile of each function is given by vector = ( , ), where (resp. ) denotes a set of CPU (resp. memory) resources allocated to . In the existing serverless platforms (e.g., AWS Lambda and Apache Open-Whisk), the resource allocation is non-preemptive and fixed, i.e., must be provisioned consistently until the function completes. We are interested in two key metrics: relative function execution time (RFET) and function throughput.
Relative Function Execution Time (RFET). We consider a FaaS platform that handles a multiple function concurrent workload. Let denote the set of functions invoked within the workload, denotes a function invocation in . At the first invocation of , FaaSRM captures the FET with resources ( , ) configured by the user and employs it as a baseline (here represents baseline). Then FaaSRM captures the FET of a further -th invocation after it completes execution. The RFET of the -th invocation is calculated as (1) Our goal is to minimize the total average RFET of a workload, which is given by¯: Since we aim to improve certain functions without degrading the performance of others, we introduce RFET to guarantee fairness between function invocations with different length of completion time.
Function Throughput. We also maximize the function throughput of a given workload. It is possible that FaaSRM harvests and allocates a big amount of resources to longrunning functions while short-running functions suffer from queuing and starving. This issue leads to low function throughput and lengthens the total duration of a workload. To avoid inappropriate resources holding and long queuing time, we make our FaaSRM aware of function throughput when provisioning workloads.
Function invocations are event-driven and arrive at the platform in real-time. The platform makes a decision on the resource allocation when serving each invocation event. We assume the resource profile is adjustable when receiving each invocation during the workload processing. Specifically at time , a function is invoked by a user. The resource profiles determines the resource provisioning for the invoked function . Existing serverless platforms require users to pre-define resources for their functions and allocate exact amount of resources when functions are invoked, which in our context the platform keeps the unchanged. However, serverless platforms can harvest resources from those over-provisioned functions and accelerate under-provisioned functions by optimizing resource allocation of each function invocation without introducing obvious performance degradation to individual functions, thus improving overall performance.

Information Collection and Embedding
When allocating resources for a function invocation, FaaSRM collects information from two levels: platform level and function level, as summarized in Table 1. Specifically, for the platform, FaaSRM captures the number of invocations remaining in the system (i.e., inflight_request_num), available CPU cores, and available memory. For the incoming function, FaaSRM queries invocation history of the function which records average CPU peak, average memory peak, average inter-arrival time, average execution time, request per second, and baseline execution time (i.e., baseline) with user-requested resources.
Once collecting such information, FaaSRM encapsulates them with a potential resource allocation option. More precisely, we embed information and the potential configuration option together into a flat state vector as input to FaaSRM agent, with the information embedding process illustrated in Figure 5.

Score Function
FaaSRM uses a score function to calculate the priority of selecting potential resource allocation options. Figure 5 visualizes the policy network of FaaSRM agent, and illustrates the workflow of how the agent selects the best allocation option based on states. At time , a function invocation arrives at the platform which has in total potential resource configuration options. After embedding procedure, FaaSRM collects a batch of the state vectors = ( 1 , . . . , , . . . , ) , where maps the state to the -th option. FaaSRM inputs to the score function. We implement the score function using two neural networks, an actor network and a critic network. The actor network computes a score , which is a scalar value mapped from the state vector representing the priority to select configuration option . Then FaaSRM applies a Softmax operation to the scores ( 1 , . . . , , . . . , ) to compute the probability of selecting option based on the priority scores, given by at time . The critic network outputs a baseline value for option , the averaged baseline value¯is calculated as which is used to reduce variance when training FaaSRM. The whole operation of policy network is end-to-end differentiable. The score function itself contains no manual feature engineering. FaaSRM agent automatically learns to compute accurate priority score of allocation options through training. More importantly, FaaSRM uses the same score function for all function invocations and all potential resource allocation options. By embedding options into state vectors, FaaSRM can distinguish between different options and use the score function to select the best option. Reusing the score function reduces the size of networks and limits the action space of FaaSRM agent significantly.

Safeguard
We design FaaSRM to improve both over-provisioned and under-provisioned functions. However, when harvesting resources from functions deemed as over-provisioned, it is possible that FaaSRM under-predicts their resource demands. The performance of functions degrades when being overharvested. We devise a safeguard mechanism atop FaaSRM to regulate decisions by avoiding decisions that may harm performance and returning harvested resources immediately when detecting a usage spike. We use this safeguard mechanism to mitigate obvious performance degradation of individual functions.
Algorithm 1 summarizes the safeguard mechanism built atop FaaSRM. We refer safeguard invocation as invoking the function with user-defined resources. When there are no previous invocations, FaaSRM triggers the safeguard invocation to obtain resource usage and calibrate the baseline mentioned in Equation 1 (lines 4-6). For further invocations, FaaSRM queries the request history of the functions, and polls the usage peak, allocation of the last invocation, and recent highest peak since last baseline calibration (lines [8][9]. FaaSRM first checks current status of the function, i.e., over-provisioned or under-provisioned (line 11). We assume functions with resource usage below 90% of user-requested Algorithm 1 Safeguard mechanism atop FaaSRM. if last_peak / user_defined <= 0.9 then 12: #Over-provisioned 13: if last_peak / last_alloc >= 0.9 then
level is over-provisioned. For over-provisioned (harvested) functions, FaaSRM then checks the usage peak of last invocation (line 13). If the usage peak approaches 90% of allocation, we suspect there may be a load spike, which could use more resources than current allocation. This triggers the safeguard invocation and baseline recalibration, FaaSRM immediately returns harvested resource to the function at the next invocation (lines [14][15]. If there is no usage spike, FaaSRM is allowed to select an allocation option from recent peak plus one unit to a user-requested level (line 17). For under-provisioned functions, FaaSRM is allowed to select from recent peak plus one unit to the maximum available level (line 21). After an allocation option is selected, FaaSRM invokes the function and forwards the invocation to worker servers for execution.
We apply the safeguard algorithm to manage CPU and memory, respectively. The results in Section 6 show that our safeguard mechanism effectively regulates decisions made by FaaSRM and protects SLOs of individual functions while improving the entire workload.

Training the RL Agent
FaaSRM training proceeds in episodes. In each episode, a series of function invocations arrive at the serverless platform, and each requires a two-dimensional action to configure CPU and memory resources. When the platform completes all function invocations, the current episode ends. Let denote the total number of invocations in an episode, and denote the wall clock time of the -th invocation. We continuously feed FaaSRM with a reward after it takes an action to handle an invocation. Concretely, we penalize FaaSRM with after taking action on the -th invocation, where denotes current number of executed invocations (i.e., function throughput), is the set of invocations that finish during the interval [ −1 , ), is the RFET of an invocation mentioned in Section 3.1, and two constant summaries for awarding good and penalizing bad actions ( (RFET<1) and (RFET>1) ). In practice, we find that taking cube root of can prevent rewards from vanishing when it goes too large. The goal of the algorithm is to maximize the expected cumulative rewards given by Hence, FaaSRM learns to minimize the average RFET of individual functions while maximizing the function throughput.
FaaSRM uses a policy gradient algorithm for training. Policy gradient methods are a class of RL algorithms that learn policies by performing gradient ascent directly on the parameters of neural networks using the rewards received during training. When updating policies, large step sizes may collapse the performance, while small step sizes may worsen the sampling efficiency. We use the Proximal Policy Optimization (PPO) algorithms [28] to ensure that FaaSRM takes appropriate step sizes during policy updates. More specifically, given a policy parameterized by , the PPO algorithm updates policies at the -th episode via where L is the surrogate advantage [27], a measure of how policy performs relative to the old policy using data from the old policy. Specifically we use the PPO-clip version of a PPO algorithm, where L is given by where is the advantage calculated as rewards subtracted by baseline values ; is a hyperparameter which restricts how far the new policy is allowed to go from the old. Intuitively, the PPO algorithm sets a range for step size of policy updates, which prevents the new policy from going too far from the old (either positive or negative). Algorithm 2 presents the training process of FaaSRM. For each episode, we record the whole set of trajectories including the states, actions, rewards, baseline values predicted by the critic network, and the logarithm probability of the actions for all invocations. After each training episode finishes, we use the collected trajectories to update the actor and critic networks.

Implementing FaaSRM
FaaSRM provides a general resource management service for functions in serverless platforms. For concreteness, we describe its implementation in the context of Apache Open-Whisk framework [4]. In this section, we briefly introduce the architecture of OpenWhisk, and describe the workflow of FaaSRM managing resources for functions in OpenWhisk system.

Integrating FaaSRM with OpenWhisk
Apache OpenWhisk is an open-source, distributed serverless platform which powers IBM Cloud Functions [15]. Figure 4 illustrates the architecture of FaaSRM integrated with Open-Whisk. OpenWhisk exposes an NGINX-based REST interface for users to interact with the platform. Users can create new functions, invoke functions, and query results of invocations via the frontend. The Frontend forwards function invocations to the Controller, which selects an Invoker (typically hosted using VMs) to execute invocations. The Load Balancer inside the Controller implements the scheduling logic by considering Invoker's health, available capacity, and infrastructure state. Once choosing an Invoker, the Controller sends the function invocation request to the selected Invoker via a Kafka-based distributed messaging component. The Invoker receives the request and executes the function using a Docker container. After finishing the function execution, the Invoker submits the result to a CouchDB-based Database and informs the Controller. Then the Controller returns the result of function executions to users synchronously or asynchronously. Here we focus on resource management for containers.
We modify the following modules of OpenWhisk to implement our resource manager: Frontend: Initially, OpenWhisk only allows users to define the memory limit of their functions and allocates CPU power proportionally based on memory. To decouple CPU and memory, we add a CPU limit and enable Frontend to take CPU and memory inputs from users. Now users are allowed to specify CPU cores and memory of their functions, and the Frontend forwards both CPU and memory limit to Controller.
Controller: The Load Balancer makes scheduling decisions for the Controller. When selecting an Invoker, the Load Balancer considers available memory of Invokers. We modify Load Balancer also to check available CPU cores of Invokers, i.e., now Load Balancer selects Invokers with enough available CPU cores and memory to execute function invocations.
Invoker: The Invoker uses a semaphore-based mechanism to control access of containers to its available memory. To manage CPU independently, we apply the identical mechanism to control access to available CPU cores.
Container: By default, OpenWhisk uses cpu-shares parameter to regulate CPU power of containers. When plenty of CPU cycles are available, all containers with cpu-shares use as much CPU as they need. While cpu-shares improves CPU utilization of Invokers, it can result in performance variation of function executions. We change the CPU parameter to cpus which restricts how many CPU cores a container can use. This is aligned with the CPU allocation policy of AWS Lambda [5]. For each function invocation, we monitor the CPU cores and memory usage of its container using  cgroups. We record the usage peak during function execution and keep it as history for FaaSRM to query.

Implementing RL Agent in FaaSRM
FaaSRM communicates with OpenWhisk Controller directly via a Key-Value (KV) Store implemented using Redis. When receiving a function invocation, the Load Balancer in the Controller first sends the current state information to the KV Store. The DRL Agent in FaaSRM then fetches the state and sends an action back to Controller. After Controller picks up the action, the Load Balancer adjusts the resource allocation of the function invocation based on the action provided by FaaSRM, and then forwards it to a chosen Invoker for execution. Because FaaSRM is a resource manager inside serverless platforms, we make FaaSRM communicate with the Controller rather than the Frontend. This reduces communication overhead and speeds up the training. We implement the prototype of FaaSRM agent using two neural networks; each with two fully-connected hidden layers. The first hidden layer has 32 neurons, and the second layer has 16 neurons; each neuron uses Tanh as its activation function. The agent of FaaSRM is implemented in Python using PyTorch [26]. FaaSRM uses multiprocessing to retrieve results of function invocations for computing rewards. The implementation of FaaSRM is lightweight as the policy network consists of 1858 parameters (12 KBs in total) because FaaSRM reuses the score function. Mapping a state to an action takes less than 50 ms.
We use the algorithm presented in Section 4.5 to train FaaSRM with 4 epochs per surrogate optimization and a 0.2 clip threshold [28]. We update the policy network parameters using the AdamW optimizer [16] with a learning rate of 0.001. For simulation, We train FaaSRM over 5000 episodes. For OpenWhisk experiment, we train FaaSRM with 500 episodes. The total time for OpenWhisk training takes about 120 hours. We restart the OpenWhisk platform before each training episode. Figure 6 shows the learning curve of FaaSRM training in simulation and OpenWhisk experiment, respectively. The descending loss trendlines in both indicate that FaaSRM gradually learns to make good resource management decisions for functions through training.

Evaluation
We conduct an extensive evaluation of FaaSRM. We evaluate FaaSRM on a simulator at scale with 1,000 unique functions sampled from Azure Functions traces [30]. We also evaluate FaaSRM on an OpenWhisk cluster with 10 realistic serverless applications. Our goal is to investigate the effectiveness and generalizability of FaaSRM under different workloads and system conditions, as FaaSRM is designed to be general and workload-agnostic. Results of simulation and OpenWhisk evaluation indicate that FaaSRM outperforms the other three baseline RMs. Compared to the default RM in OpenWhisk, FaaSRM reduces the execution time of 98% of function invocations by 87.60% and 35.81% for the same workload, in simulation and OpenWhisk, respectively. On the OpenWhisk cluster, FaaSRM harvests idle resources from 38.78% of function invocations while accelerating 39.18%.

Methodology
For evaluating performance of FaaSRM with different workload types, we use different real-world trace samples from the Azure Functions traces [30] in both simulating and Open-Whisk evaluation, which contains execution times, memory sizes, and invocation-timestamps for more than 50,000 unique functions.
In two evaluations, we compare FaaSRM with three baseline RMs: (1) Fixed RM: the default RM employed by most existing serverless platforms. Fixed RM requires users to pre-define memory for their functions and then consistently provisions the exact amount of memory. CPU power is then allocated proportionally to functions based on the memory. We assume users pre-configure their memory and make no changes after invocation starts. (2) Greedy RM: a greedy RM that proactively optimizes resource allocation by harvesting idle resources and supplements resources based on fixed size of steps. For each function invocation, Greedy RM also queries the same history that FaaSRM accesses. Rather than using RL predictions, Greedy RM increases or decreases resources based on a fine-tuned, fixed step size. In our implementation, we set the step size to be 1 core in CPU management and 64 MBs in memory. (3) ENSURE: the proposed RM in [32], which manages CPU resources for serverless platforms at the function level. ENSURE dynamically adjusts CPU power for each function only when detecting performance degradation. However, ENSURE doesn't provide memory management, i.e., function memory stays fixed as user-requested. We implement the CPU management policy of ENSURE as one of our baselines.

Simulation at Scale
We first evaluate FaaSRM in a simulative serverless computing environment based on OpenAI Gym [25], an open-source library for evaluating RL algorithms. To build a simulating framework that closes to real serverless platforms, we refer   to various features of OpenWhisk. In the simulator, we mimic the features of containerization or sandbox techniques in a real serverless platform. Whenever users pick new memory limits for their functions, new containers have to be launched, thus leading to cold starts [18]. We implement APIs for defining a customized simulative cluster. The cluster can be configured with arbitrary numbers of servers, and each server is enabled with features of a realistic serverless environment, such as temporary function dependency caching for warm starts and message queuing for high volumes of invocation bursts. The function placement problem is not discussed in FaaSRM, for which we adopt the default hashing algorithm of OpenWhisk load balancer. We configure our simulating cluster with 20 worker servers, each with 8 CPU cores and 2 GBs memory available for functions. Each function has access to 8 CPU cores and 512 MBs memory at most. Workloads: We randomly sample two workload sets consisting of over 16,000 unique functions from Azure Functions traces. Table 2 (set SIM-train and SIM-test) depicts the characteristics of workload sets we use in the simulating evaluation for training and testing, respectively. In simulation, we examine the performance of FaaSRM when serving serverless functions at scale.
Results: In the simulation, FaaSRM outperforms other baseline RMs by achieving a minimal average RFET of the workload with 0.76, whereas Fixed RM, Greedy RM, ENSURE are of 1.00, 1.49, and 0.91, respectively. Due to some sampling issues of traces, FixedRM has few outliers that RFETs are not 1. Figure 7 presents the RFETs of 85,470 function invocations processed by four RMs. Fixed RM has no performance degradation but also no resource adjustment. Greedy RM experiences some serious SLO violations while harvesting too much resource. ENSURE also violates SLOs of invocations even only adjusting CPU resources. In contrast, FaaSRM rationally harvests idle resources from over-provisioned invocations and provides under-provisioned with significant acceleration. Figure 8 shows the cumulative distribution function (CDF) of function execution times of all the invocations. FaaSRM processes most of function invocations faster than other baseline RMs. FaaSRM completes the 98% percentile of invocations with execution time less than 15 seconds, whereas Fixed RM, Greedy RM, and ENSURE complete execution with 121, 76, and 48 seconds, respectively.    In simulation, FaaSRM outperforms other baseline RMs with a minimal average RFET by achieving fairness between invocations within a workload, i.e., FaaSRM maintains SLO of each invocation and improves as many invocations as possible. We then evaluate FaaSRM and other baselines in a real-world serverless platform, OpenWhisk, with realistic serverless applications.

OpenWhisk Evaluation
We deploy and evaluate FaaSRM on an OpenWhisk cluster with 13 servers. One server hosts the REST frontend, API gateway, and Redis services; one backend server hosts the Controller, Distributed Messaging, and Database services; one server hosts FaaSRM agent; and the remaining 10 servers are Invokers for executing functions. The server hosting FaaSRM agent has 16 Intel Xeon Skylake CPU cores and 64 GB memory, whereas each of the other 12 servers has 8 Intel Xeon Skylake CPU cores and 32 GB memory. Each Invoker provided 8 CPU cores and 2 GB RAM for individual function executions. Each function can be configured with 8 CPU cores and 512 MBs of RAM at most. We use a window of 100 ms to monitor the CPU and memory usage of function invocations inside containers.
Workloads: We randomly sample another two invocation sets for OpenWhisk evaluation. Table 2 depicts the two invocation sets (OW-train and OW-test) used in the Open-Whisk evaluation. We use a scaled-down version of the invocation traces, i.e., we assume the invocation trace is based on seconds rather than minutes. This re-scaling increases the intensity of workloads while speeding up FaaSRM Open-Whisk training by reducing the total workload duration. We employ 10 real-world applications from three serverless benchmark suites: SeBS [7], ServerlessBench [36], and ENSURE-workloads [32]. Table 3 describes the characteristics of 10 serverless applications, including average CPU and memory usage peak during execution, and average cold and warm duration. We set 8 CPU cores and 512 MBs for VP, IR, and DV as their initial user-defined allocation. For other functions, we set 4 CPU cores and 256 MBs as their initial user-defined allocation.
Results: For the testing workload set in OpenWhisk evaluation, FaaSRM outperforms other baseline RMs by achieving an average RFET of the workload with 0.89, whereas Fixed RM, Greedy RM, ENSURE are of 1.09, 0.93, and 1.12, respectively. Notice that in OpenWhisk experiment, the average RFET of Fixed RM is not strictly 1.0 due to performance variation. Figure 9 shows the RFETs of 268 function invocations processed by four RMs. Fixed RM has no resource adjustment. Both Greedy RM and ENSURE severely violate SLOs of some invocations while harvesting resources. In contrast to the perfect SLOs of invocations in simulation, evaluating FaaSRM in OpenWhisk occurs some performance variation. However, FaaSRM still achieves a minimal average RFET and maintains the 98% percentile of RFETs below 1.28 for all invocations, whereas Fixed RM, Greedy RM, and ENSURE are 1.27, 2.52, and 5.2, respectively. Figure 10 shows the cumulative distribution function (CDF) of function execution times of all the invocations. FaaSRM processes most of function invocations faster than other baseline RMs. FaaSRM completes the 98% percentile of invocations with execution time less than 16.91 seconds, whereas Fixed RM, Greedy RM and ENSURE complete execution with 32.16, 25.80, and 27.39 seconds, respectively.
We conduct extensive experiments in both simulation and OpenWhisk evaluation. FaaSRM outperforms other baseline RMs with minimal average RFET by achieving fairness between invocations within a workload, i.e., FaaSRM guarantees SLOs of individual invocations and improves as many invocations as possible. Hence, FaaSRM harvests idle resources from over-provisioned functions and accelerates under-provisioned functions with supplementary resources while preventing any functions from apparent performance degradation.

Related Work
Cold start. Recent research works propose different policies based on serverless computing's unique characteristics, such as fine-grained resource granularity and event-driven. The paper [30] indicates that most serverless applications have exigent patterns in invocation frequency, duration, or resource consumption. They propose a diagram resource management policy based on Azure Functions traces to predict pre-warming and keep-alive windows of containers for serverless applications. FaaSProfiler [29] investigates architectural implications of serverless computing from a provider view. FaasCache [10] uses caching techniques to optimize cold starts in serverless environment. Resource management. Research has been conducted on VM resource management in traditional clouds for years. Recently, SmartHarvest [35] proposes a VM resource harvesting algorithm using online learning. Unlike FaaSRM, which uses harvested resources to accelerate function executions, SmartHarvest offers a new low-priority VM service using harvested resources. Directly replacing FaaSRM with SmartHarvest is not feasible as SmartHarvest is not designed for serverless computing. Spock [13] proposes a serverlessbased VM scaling system to improve SLOs and reduce costs. For resource management in serverless, [20] and [32] both aim to automatically adjust CPU resource when detecting performance degradation during function executions, which help mitigate the issue of resource over-provisioning. In contrast to [20] and [32], which only focus on CPU, FaaSRM manages CPU and memory resources independently. In [19], they propose a centralized scheduler for serverless platforms that assigns each CPU core of worker servers to CPU cores of scheduler servers for fine-grained core-to-core management. FaaSRM focuses on resource allocation rather than scheduling or scaling. Reinforcement learning. Siren [14] adopts DRL techniques to dynamically invoke functions for distributed machine learning with a serverless architecture. Our work FaaSRM leverages DRL to improve the platform itself rather than serverless applications. Decima [22] leverages RL to schedule DAG jobs for data processing clusters. Metis [34] proposes a scheduler to schedule long-running applications in large container clusters.

Concluding Remarks
This paper proposed a new resource manager, FaaSRM, which harvests idle resources from over-provisioned functions and accelerates under-provisioned functions with supplementary resources. Given realistic serverless workloads, FaaSRM can improve most of function invocations while safely harvesting idle resources using the reinforcement learning and the safeguard mechanism. Results of simulation and experiments on the OpenWhisk cluster demonstrate that FaaSRM outperforms other baseline RMs. Compared to the default RM in OpenWhisk, FaaSRM reduces the execution time of 98% function invocations by 87.60% and 35.81% for the same workload, in simulation and OpenWhisk, respectively. In OpenWhisk evaluation, FaaSRM harvests idle resources from 38.78% of function invocations while accelerating 39.18%.