{'fraction_non_alphanumeric': 0.03316611649728381, 'fraction_numerical': 0.015764302237933094, 'mean_word_length': 5.379290333278063, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scheduling IoT Applications in Edge and Fog Computing Environments: A Taxonomy and Future Directions

Fog computing, as a distributed paradigm, offers cloud-like services at the edge of the network with low latency and high-access bandwidth to support a diverse range of IoT application scenarios. To fully utilize the potential of this computing paradigm, scalable, adaptive, and accurate scheduling mechanisms and algorithms are required to efficiently capture the dynamics and requirements of users, IoT applications, environmental properties, and optimization targets. This paper presents a taxonomy of recent literature on scheduling IoT applications in Fog computing. Based on our new classification schemes, current works in the literature are analyzed, research gaps of each category are identified, and respective future directions are described.


INTRODUCTION
The Internet of Things (IoT) paradigm has become an integral part of our daily life, thanks to the continuous advancements of hardware and software technologies and ubiquitous access to the Internet. The IoT concept spans a diverse range of application areas such as smart city, industry, transportation, smart home, entertainment, and healthcare, in which context-aware entities (e.g., sensors) can communicate together without any temporal or spatial constraints [41,52]. Thus, it has shaped a new interaction model among different real-world entities, bringing forward new challenges and opportunities. According to Business Insider [1] and International Data Corporation (IDC) [2], 41 Billion active IoT devices will be connected to the Internet by 2027, generating more than 73 Zettabytes of data. The real power of IoT resides in collecting and analyzing the data circulating in the environment [63], while the majority of IoT devices are equipped with a constrained battery, computing, storage, and communication units, preventing the efficient execution of IoT applications and data analysis on time. Thus, data should be forwarded to surrogate servers for processing and storage. The processing, storage, and transmission of this gigantic amount of IoT data require special considerations while considering a diverse range of IoT applications.
CSs for the execution of IoT applications, making the scheduling problem even more complex. However, resource-sharing among FSs is less resilient than CSs due to spatial constraints and high heterogeneity in deployed operating systems, standards, and protocols, just to mention a few [67,76,79]. In addition, FSs are more exposed to end-users which makes them potentially less-secured compared to CSs [126]. Besides, the geo-distribution of resources and the IoT data hosted and shared on different servers may also impose privacy implications. As many IoT users may share personal information in Fog computing environments, adversaries can gain access to this shared information [121].
• Challenges related to optimizing parameters Optimizing the performance of IoT applications running in Fog computing environments depends on numerous parameters such as the main goal of each IoT application, the capabilities of IoT devices, servers properties, networking characteristics, and the imposed constraints. Optimizing the performance of even one IoT application in such a heterogeneous environment with numerous contributing parameters is complex, while multiple IoT applications with different parameters and goals further complicate the problem. • Challenges related to decision engines: Decision engines are responsible to collect all contextual information and schedule IoT applications. Based on the context of IoT applications and environmental parameters, these decision engines may use different optimization modeling [68]. Besides, there are several placement techniques to solve these optimization problems. However, considering the types of IoT application scenarios and the number and types of contributing parameters, different placement techniques lead to completely different performance gain [39,129]. For example, some placement techniques result in high-accuracy decisions while their decision time takes a long time. However, some other techniques find acceptable solutions with shorter decision time. Moreover, the decision engines can be equipped with several advanced features such as mobility support and failure recovery, enabling them to work in more complex environments. • Challenges related to real-world performance evaluation: The lack of global Fog service providers offering infrastructure on pay-as-you-go models like commercial Cloud platforms such as Microsoft Azure and Amazon Web Services (AWS) pushes researchers to set up small-scale Edge/Fog computing environments [77]. The real-world performance evaluation of IoT applications and different placement techniques in Fog computing is not as straightforward as Cloud computing since the management of distributed FSs incurs extra effort and cost, specifically in large-scale scenarios. Besides, the modification and tuning of system parameters during the experiments are time-consuming. Hence, while real-world implementations are the most accurate approach for the performance evaluation of the systems, it is not always feasible, specifically in large-scale scenarios.

Motivation of Research.
Numerous techniques for scheduling IoT applications in Fog computing environments have been developed to address the above-mentioned challenges. Several works focused on the structure of IoT applications and how these parameters affect the scheduling [29,79,101] while some other techniques mainly focus on environmental parameters of Fog computing, such as the effect of hierarchical Fog layers on the scheduling of IoT applications [38,62]. Besides, several techniques focus on defining specific optimization models to formulate the effect of different parameters such as FSs' computing resources, networking protocols, and IoT devices characteristics, just to mention a few [68]. Moreover, several works have proposed different placement techniques to find an acceptable solution for the optimization problem [5,51,78] while some other techniques consider mobility management [26,38,131] and failure recovery [8,40]. All these perspectives directly affect the scheduling problem, especially, when designing decision  engines. These perspectives should be simultaneously considered when studying and evaluating each proposal. However, only a few works in the literature have identified the scheduling challenges that directly affect the designing and evaluation of decision engines in Fog computing environments and accordingly categorized proposed works in the literature. Thus, we identify five important perspectives regarding scheduling IoT applications in Fog computing environments, as shown in Fig. 2, namely application structure, environmental architecture, optimization modeling, decision engines' characteristics, and performance evaluation. Fig. 3 depicts the relationships among identified perspectives. The features of application structure and environmental architecture help define the optimization characteristic and formulate the problem. Then, an efficient decision engine is required to effectively solve the optimization problem. Besides, the performance of the decision engine should be monitored and evaluated based on the main goal of optimization for the target applications and environment. Considering each perspective, we present a taxonomy and review the existing proposals. Finally, based on the studied works, we identify the research gaps in each perspective and discuss possible solutions. The main contributions of this work are: • We review the recent literature on scheduling IoT applications in Fog computing from application structure, environmental architecture, optimization modeling, decision engine characteristics, and performance evaluation perspectives and propose separate taxonomies. • We identify research gaps of scheduling IoT applications in Fog computing considering each perspective. • We present several guidelines for designing a scheduling technique in Fog computing paradigm. • We identify and discuss several future directions to help researchers advance Fog computing paradigm.

Paper Organization
The rest of the paper is organized as follows. The existing related surveys and taxonomies on scheduling IoT applications in Fog computing environments are studied and compared with ours in Section 2. Section 3 presents a taxonomy and overview of the IoT applications' structure. Section 4 introduces a taxonomy on environmental properties of resources in Fog computing environments and studies the existing works accordingly. In Section 5, a taxonomy of optimization characteristics of problems in Fog computing environments is introduced. Section 6 identifies the important aspects of decision engines and presents a taxonomy of decision engines for scheduling IoT applications. Section 7 demonstrates the approaches and metrics used for the evaluation of scheduling strategies in Fog computing. Section 8 presents a guideline for designing a scheduling technique. According to identified research gaps, Section 9 provides several future research directions. Finally, Section 10 concludes this survey.

RELATED SURVEYS
In the context of Fog computing, surveys targeted different aspects of Fog computing, such as security [112,121,126,154], smart cities [105], live migration techniques [99], existing software and hardware [107], deep learning applications [139], healthcare [66], and general surveys studied the Fog computing paradigm, its scope, architectures, and recent trends [10, 44, 52, 91-93, 96, 122]. Also, some surveys mainly discussed resource management, application management, and scheduling in the context of Fog computing, such as [3,34,60,68,79,86,113,118,152], that we discuss and compare them with ours. Aazam et al. [3] reviewed enabling technologies and research opportunities in Fog computing environments alongside studying computation offloading techniques in different domains such as Fog, Cloud, and IoT. Hong et al. [48] and Ghobaei-Arani [34] studied resource management approaches in Fog computing environments and discussed the main challenges for resource management. Yousefpour et al. [152] discussed the main features of the Fog computing paradigm and compared it with other related computing paradigms such as Edge and Cloud computing. Besides, it studied the foundations, frameworks, resource management, software, and tools proposed in Fog computing. Mahmud et al. [79] mainly discussed the application management and maintenance in Fog computing and proposed a taxonomy accordingly. Salaht et al. [113] presented a survey of current research conducted on service placement problems in Fog Computing and categorized these techniques. Shakarami et al. [118] studied machine learning-based computation offloading approaches while Adhikari et al. presented the type and applications of nature-inspired algorithms in the Edge computing paradigm. Martinez et al. [86] mainly focused on designing and evaluating Fog computing systems and frameworks. Lin et al. [68] and Sonkoly et al. [123] mainly studied and categorized different approaches for modeling the resources and communication types for computation offloading in Edge computing. Finally, Islam et al. [60] proposed a taxonomy for context-aware scheduling in Fog computing and surveyed the related techniques in terms of contextual information such as user and networking characteristics. Table 1 summarizes the characteristics of related surveys and provides a qualitative comparison with our work. The proper scheduling of IoT applications in Fog computing environments can be viewed from different perspectives, such as application structure, environmental architecture, optimization modeling, and the features of decision engines. Besides, the performance of scheduling techniques should be continuously evaluated to offer the best performance. As depicted in Table 1, the existing surveys barely study and provide comprehensive taxonomy for the above-mentioned 3.5 [152] 3 [34] 2.5 [113] 2 [79] 1.5 [118] 1.5 [86] 1.5 [68] 1.5 [60] 1 [123] 0.5 [7] 0.5 This Survey Current : Full Cover, : Partial Cover, : Does Not Cover perspectives. In this work, we identify the main parameters of each perspective and present a taxonomy accordingly. Moreover, we identify related research gaps and provide future directions to improve the Fog computing paradigm.

APPLICATION STRUCTURE
The primary goal of Fog computing is to offer efficient and high-quality service to users with heterogeneous applications and requirements. Hence, service providers require a comprehensive understanding of each IoT application structure (e.g., workload model and latency requirements) to better capture its complexities, perform efficient scheduling and resource management, and offer high-quality service to the users. Also, when designing the architecture of each IoT application, dynamics, constraints, and complexities of resources in Fog computing should be carefully considered to exploit the potential of this paradigm. Fig. 4 presents a taxonomy and main elements of IoT application structure, described below.

Architectural Design
The logic of IoT applications can be implemented in different ways. To illustrate, operations of a Video Optical Character Recognition (VideoOCR) such as capturing frames, similarity check, and text extraction can be implemented as a single encapsulated program or as a set of interdependent components [24]. Hence, according to the granularity level of applications, their distribution, and coupling intensity, the architectural design of applications can be classified into four types: 3.1.1 Monolithic. It encapsulates the complete logic of an application as a single component or program. The parallel execution of these applications can be obtained using multi processing approaches [79]. In the context of Fog computing, several works such as [13,20,84,89] have considered monolithic applications.
3.1.2 Independent. These applications require the execution of a set of independent tasks or components for the complete execution of the application. The constituent parts of these applications can be simultaneously executed on different FSs or CSs. Several works such as [9,46,55,104] discuss applications with independent components or tasks in the Fog literature.

Modular.
Each modular application is composed of a set of dependent tasks or components. While constituent parts of each application can be distributed over several FSs or CSs for parallel execution, there are some constraints for the execution of tasks based on their data-flow dependency model. Several works in the literature such as [39,70,72,103] discuss modular applications.  [24,25,38,137] have considered loosely-coupled applications.

Granularity-based Heterogeneity
Tasks within an IoT application have different properties such as computation size, input size, output size, and deadline. These features affect the scheduling complexity, where identifying the dynamics of applications with heterogeneous task properties requires further effort. Accordingly, we categorize IoT applications based on their granularity-level specifications to 1) heterogeneous such as [47,70] or 2) homogeneous such as [55,153].

Workload Model
The workload model specifies the data architecture of an application, which can be broadly divided into two categories for IoT applications: 3.3.1 Stream/Realtime. In this category, the data should be processed by the servers as soon as it was generated (i.e., real-time), and hence, the data usually require relatively simple transformation or computation. Several works such as [22,65,82] discuss stream workload for IoT applications.

Batch.
In batch processing, the input data of an application is usually bundled for processing. However, contrary to heavy batch processing models, IoT applications often use micro-batches to provide a near-realtime experience. In the literature, several works such as [19,45,138] consider batch workload for the applications.

Communication-Computation Ratio (CCR)
Each IoT application, regardless of its architecture, contains some amount of input data for transmission and computational load for processing. These properties can significantly affect the scheduling decision to find proper FSs or CSs for an application. The CCR defines whether an application on average is more 1) computation-intensive [42,101,132] or 2) communication-intensive [13,49,116]. Besides, some works consider a range of applications to cover both computationintensive and communication-intensive applications, to which we refer as 3) hybrid [16,40].

Discussion
In this section, we discuss the effects of identified application structure's elements on the decision engine and describe the lessons that we have learned. Besides, we identify several research gaps accordingly. Table 2 summarizes the characteristics related to IoT application structure in Fog computing.
3.5.1 Effects on the decision engine. The application structure properties affect the decision engine in various aspects, as briefly described below.
1. Architectural design: It defines the number of tasks/modules and their respective dependency within a single application. Hence, as the number of tasks/modules per application increases, the problem space significantly increases. Considering an application with number of tasks and possible candidate configuration per task, the Time Complexity (TC) of finding the optimal solution is ( ). Besides, the dependency of tasks within an application imposes hard constraints on the problem, which further increases the complexity. Thus, finding the optimal solution for the scheduling of applications becomes very time-consuming, and the design of an efficient placement technique to serve applications in a timely manner remain an important yet challenging problem.

2.
Granularity-based heterogeneity: It shows the corresponding properties of each task/module within an application and plays a principal role in identifying the dynamics of applications. One of the most important features of decision engines is their adaptability and their capability to extract the complex dynamics of applications so that the decision engine can receive diverse types of applications' requests. Since applications with heterogeneous granularity-based properties have higher dynamics' complexity, the decision engines designed for this application category should support high adaptability.

3.
Workload model and CCR: These elements provide insightful information regarding the input data architecture of the application and its behavior in the runtime. Accordingly, the decision engine may define different priority queues for incoming requests based on their workload model and CCR to provide higher QoS for the users. For example, applications with real-time workload types and communication-intensive CCR may have higher priority for the placement on servers closer to the IoT devices than computation-intensive applications that are not real-time.
3.5.2 Lessons learned. Our findings regarding the IoT application structure in the surveyed works are briefly described in what follows: 1. Almost 70% of the surveyed works have overlooked studying the dependency model of tasks within an application and selected either the independent or monolithic design. The rest of the works consider dependency among tasks of an application in different models (i.e., sequential, parallel, or hybrid dependency). Moreover, only about 10% of the studied works consider microservices in their application design.

2.
The most realistic assumption for the granular properties of each task/module is heterogeneous (i.e., heterogeneous input size, output size, and computation size). Almost 85% of the studied works consider heterogeneous properties for each task/module, while around 15% of the works consider the homogeneous properties for the tasks/modules.
3. The workload model and CCR in each proposal depend on the targeted application scenarios. Almost 55% of the works did not study the CCR, or the required information to obtain the CCR (i.e., computation size of tasks, average data size) was not mentioned. Among the rest of the works, computation-intensive, communication-intensive, and hybrid CCR form roughly 25%, 15%, and 5% of proposals respectively.

Research Gaps.
We have identified several open issues for further investigation, that are discussed below: 1. According to Alibaba's data of 4 million applications, more than 75% of the applications consist of dependent tasks [155]. However, only around 30% of the recent works surveyed in this study consider applications with dependent tasks (i.e., modular or loosely-coupled), showing further investigation is required to identify the dynamics of these types of complex applications.

2.
Although the microservice-based applications can significantly benefit the IoT scenarios, only a few works such as [38,137] have studied the scheduling and migration of microservices in Edge/Fog computing environments. So, further investigation is required to study the behavior of microservice-based applications with different resource management techniques.
3. Modular or loosely-coupled IoT applications can be distributed over different FSs or CSs for parallel execution. However, several works such as [78] statically assign components of an application on pre-defined FSs or CSs and only schedule one or two remaining components. Hence, the best placement configuration of applications based on the current dynamics of the system cannot be investigated, leading to diminished performance gain.

4.
When the number of IoT applications increases, there is a high probability that application requests with different workload models are submitted to the system. However, none of the studied works in the literature consider applications with different workload models and how they may mutually affect each other in terms of performance.

5.
Due to the high heterogeneity of IoT applications in Fog, applications with diverse CCR may be submitted for processing, requiring special consideration such as networking and prioritization. Although there are only a few recent works such as [22,39] that consider hybrid CCR, most of the recent works target one of the computation-intensive or communication-intensive applications.

ENVIRONMENTAL ARCHITECTURE
The configuration and properties of IoT devices and resource providers directly affect the complexity and dynamics of scheduling IoT applications. To illustrate, as the number of resource providers increases, heterogeneity in the system also grows as a positive factor, while the complexity of making a decision also increases that may negatively affect the process of making decisions. In this section, we classify the environmental architecture properties, as depicted in Fig. 5, into the following categories:

Tiering Model
IoT devices and resource providers can be conceptually organized in different tiers based on their proximity to users and resources, described below: 4.1.1 Two-Tier. In this resource organization, IoT devices are situated at the bottom-most layer and resource providers are placed at the edge of the network in the proximity of IoT devices (i.e., Edge computing). Several works use two-tier resource organization such as [55,69,84,125].

Three-Tier.
Compared to two-tier model, this model also uses CSs at the highest-most layer to support edge resources (i.e., Fog computing). Several works considered three-tier model in the literature such as [59,65,108,116].

Many-Tier.
In many-tier resource organizations, IoT devices and CSs are situated at the bottom-most and highest-most tiers respectively, while FSs are placed in between through several tiers (i.e., hierarchical Fog computing). In the literature, several works have considered many-tier model such as [31,38,83,111].

IoT Devices
IoT devices can play two roles; as service requester and/or resource provider. When they act as service requesters, still they can execute a portion of their tasks or components based on their available resources. Moreover, IoT devices can simultaneously play these different roles. We study the properties of IoT devices from the following perspectives: 4.2.1 Number. The higher number of IoT devices (either as service requester or service provider), the higher complexity of the scheduling problem. Some works only consider single IoT device in the environment such as [70,97,138] while other works consider multiple IoT devices simultaneously such as [50,100,133].

Type.
The type of IoT devices help us understand the amount of resources, capabilities, and constraints of these devices. The IoT devices used in the current literature can be broadly classified into three categories, namely 1) mobile devices (MD) which are mostly considered as smartphones or tablets [25,102,119], 2) Vehicles [137,140,148], and 3) General devices containing a set of IoT devices, ranging from small sensors to drones [39,84,120].

Heterogeneity.
We also study the resources of IoT devices and their request types, and classify proposals into 1) heterogeneous where IoT devices have various resources and different request types and sizes such as [31,72,145] or 2) homogeneous where the resources of IoT devices are the same or they have the same request type and size such as [70,138,149].

Fog Servers (FSs)
FSs usually act as resource providers for IoT devices. The environmental properties of FSs can be classified based on the following criteria: 4.3.1 Number. Similar to IoT devices, we classify the number of FSs in the environment into 1) Single and 2) Multiple. The complexity and dynamics of system in surveyed works that have considered only single FS such as [53,54,110] is simpler to the works that have considered multiple FSs such as [28,127,146].

Type.
The type of FSs acting as service provider in the Fog computing environment ranges from IoT devices with additional resources to resource-rich data centers. Several works have considered a specific type of FS and discuss their properties in their works such as 1) Base Station (BS) and Macro-cell Station (MS) [18,127,138], 2) femtocells [19,32,37], 3) Rpi [30] and 4) access points (AP) [27,88]. Moreover, several works consider 5) general FSs containing a set of FSs with different types such as [31,94].

Heterogeneity.
We study the FSs' resources and classified works based on their heterogeneity into 1) heterogeneous and 2) homogeneous accordingly. Many works have considered heterogeneous resources for FSs [9,39,46,102] while some works consider homogeneous resources for FSs [54,73,134].

Cooperation.
Compared to CSs, each FS has fewer resources and it may not be able to satisfy the requirements of IoT applications. Cooperation among FSs helps augmenting their resources and providing service for demanding IoT applications. We classify proposals based on their cooperation among FSs into 1) intra-tier where FSs of same tier collaborate to satisfy users' requests [24,25,25,109] and 2) inter-tier where FSs of different layers also collaborate for the execution of IoT one application [38,39].

Cloud Servers (CSs)
The environmental properties of CSs can be classified based on the following criteria: The current literature based on the CSs' number can be divided into 1) single and 2) multiple categories. Majority of works only consider one CDC as resource provider (either as a central entity with aggregated resources or different number of VMs) to support FSs such as [14,50,59,65]. In real-world environment, different CDCs are available which can provide services with different QoS for multiple applications. Some works such as [35,39,40,132] have considered multiple CDCs with heterogeneous CSs in the literature.

Cooperation.
Among the works considered multi CDCs, we study either CSs from different CDCs are configured to collaboratively execute an IoT application or not. In the literature, some works such as [35,40,104] have considered collaborative multi CDCs scenarios.

Discussion
In this section, we discuss the effects of identified environmental architecture's elements on the decision engine and describe the lessons that we have learned. Besides, we identify several research gaps accordingly. Table 3 provides a summary of properties related to environmental architecture in Fog computing.

4.5.1
Effects on the decision engine. The elements of environmental architecture affect the decision engine in various aspects, as briefly described below.
1. Tiering: It represents the organization of end-users' devices and resources in the computing environment. Considering the properties of resources in different tiers, it helps find the most suitable deployment layer for the decision engine to efficiently serve IoT applications' requests with a wide variety of requirements. For example, to serve real-time IoT applications with low startup time requirements, the most suitable deployment layer in the three-tier model is the lowest-level Fog layer.

2.
IoT devices: The number of IoT devices directly relates to the number of incoming requests to decision engines. It affects the admission control of decision engines. The type of IoT devices provides contextual information about the number of resources and intrinsic properties of the IoT devices that are important for the decision engine. For example, the MD type not only states that the IoT device does not have significant computing resources, but also presents that the device has mobility features. Thus, the IoT device type affects the advanced features of the decision engine, such as mobility, and also specifies whether the IoT devices can serve one or several tasks/modules of IoT applications or not.
3. Fog and Cloud servers: The number of available servers directly affects the TC of the scheduling problem. Considering an application with number of tasks and possible candidate configuration per task, the TC of finding the optimal solution is ( ). Hence, it directly affects the choice of placement technique and scalability feature of the decision engine. As the problem space increases, a suitable decision engine should be selected to solve the scheduling problem. Moreover, the type and heterogeneity of resources provide further contextual information for the decision-making, such as the number of resources, networking characteristics, and resource constraints, just to mention a few.

Lessons learned.
Our findings regarding the environmental architecture in the surveyed works are briefly described in what follows: 1. Almost 60% of works consider the three-tier model and many-tier models for the organization of end-users and resources. Not only do these works consider real-time applications, but also some of them assume both real-time and computation-intensive applications, such as [16,39,124]. This is mainly because these works use CSs as a backup plan for more computation-intensive applications or when the number of incoming IoT requests increases and the FSs cannot solely manage the incoming requests. Moreover, nearly 40% of surveyed works assume a two-tier model for the organization of end-users and resources. These works mostly assume real-time workload type and communication-intensive applications for the deployment on Edge servers, such as [13,22,53,110].

2.
In the surveyed works, almost 90% of the works considered an environment with multiple IoT devices, while 10% of works only focused on a single IoT device. When the number of IoT devices increases, the diversity of IoT applications and heterogeneity of their tasks also increase accordingly. Moreover, the greater number of works assume IoT devices as general devices with sensors, actuators, and diverse application requests. In contrast, some works targeted a specific IoT devices such as mobile devices and vehicles with almost 30% and 10% of proposals, respectively. These proposals studied other contextual information of targeted IoT devices in detail such as mobility [138], energy consumption [140], and networking characteristics [109,137]. Finally, about 90% of works have studied IoT devices with heterogeneous properties and diverse application request types, which are the closest scenario to real-world computing environments.
3. Regarding Fog resources, almost 90% of the proposals consider multiple FSs in the environment. However, only 40% of the current literature has considered any cooperation model among FSs. There is a high probability that a single FS cannot solely manage the execution of several incoming resources due to its limited resources. Also, sending partial/complete applications' tasks to the Cloud may negatively affect IoT devices' response time and energy consumption, especially for real-time IoT applications. Thus, cooperation among FSs is of paramount importance that can lead to the execution of IoT applications with better performance and QoS. Considering the type of the FSs, about 60% of the studied literature considered general FSs. The rest of the works studied a specific type of FSs and tried to involve their contextual information in the scheduling process of IoT applications, such as networking characteristics [9]. Moreover, some works considered IoT devices can simultaneously play different roles in the computing environments (i.e., service requester and service provider) such as [132,140,157].
4. In the current literature, around 60% of the works consider CSs as computing resources in the environment. However, only in 8% of these works multiple Cloud service providers (i.e., multi-Cloud), their communication, and interactions are studied, such as [39,40,104,132].

Research Gaps.
We have identified several open issues for further investigation, as discussed below: 1. Hierarchical Fog computing (i.e., multi-tier) has not been thoroughly considered by researchers. Only a few works (almost 5%) consider the organization of resources in the multi-tier environment, and most have focused on the heterogeneity of resources among different tiers. However, these works have not considered the heterogeneity of communication protocols and latency in multi-tier environments.

2.
In the literature, several works have considered abstract CDC as a central unit with huge computing capacity [80], while in reality, CDCs contain several CSs hosting computing instances. Such assumptions affect the computing and communication time in simulation studies.

3.
One of the main advantages of Fog computing is providing heterogeneous FSs in IoT devices' vicinity to collaboratively serve applications. However, many works have not considered cooperation among FSs. In this case, due to the limited computing and communication resources of each FS and a large number of IoT requests, the serving FS may become a bottleneck which negatively affects the response time and QoS [37]. Besides, in uncooperative scenarios, the overloaded FS forwards requests to CSs, incurring higher latency. Hence, cooperative Fog computing, associated protocols, and constraints require further investigation for different IoT application scenarios.

OPTIMIZATION CHARACTERISTICS
Considering the application structure, environmental parameters, and the target objectives, each proposal formulates the problem of scheduling IoT applications in Fog computing. Optimization parameters directly affect the selection process and properties of suitable decision engines. Fig. 6 presents the principal elements in optimization characteristics, as described in what follows:

Main Perspective
The proposals in the literature can be divided into three categories based on their main optimization goal, namely IoT devices/users, system, and hybrid, which are described in the following: 5.1.1 IoT devices/users. The main perspective of several proposals is to satisfy the requirements of IoT applications such as minimizing their execution time and energy consumption of IoT devices, or improving user experience in terms of QoS and Quality of Experinece (QoE). Several works have considered IoT perspective for the optimization such as [13,73,114,114,124,134].

System.
The main perspective of this category is to improve the efficiency of resource providers such as minimizing their energy consumption, improving resource utilization, and maximizing the monetary profit [12,19,28,50]. Hence, these works often assume IoT devices with very limited limited computational resources that transfer sensed data to the surrogate servers for processing and storage.

Hybrid.
Some proposals targeted optimizing the parameters of both IoT devices/users and resource providers, referred to as hybrid optimization [55,82,98,144]. In these works, IoT devices have some computational resources to serve their partial/complete tasks. However, they may send their requests to other surrogate servers if overall global parameters of IoT devices and systems can be optimized.

Objective Number
According to the number of main optimization objectives of each proposal, we classify the current literature into 1) single objective and 2) multi objective proposals. Multi objective proposals consider several parameters to simultaneously optimize them, incurring higher complexity. In the literature, several proposals targeted single objective optimization such as [56,69,114,157], while other proposals try to optimize several parameters such as [38,46,95,145].

Parameters
According to the main objectives and the nature of optimization parameters in the literature, we categorize these parameters into the following categories:

Hybrid.
Several works also have considered a set of optimization parameters, referred as hybrid. These works use any combination of above-mentioned parameters simultaneously [39,40,46,134].

Problem Modeling
Considering the main goal and optimization parameters, the optimization problem can be modeled/formulated. Considering surveyed literature in terms of the problem modeling approach, we classify the works into the following categories:

Mixed
Integer Linear Programming (MILP). In these problems, only some of the variables are constrained to be integers, while other variables are allowed to be non-integers. Also, the objective function and equations are linear. Several works have modelled their problem as an MILP such as [55,70,104,145].

Mixed Integer Non-Linear Programming (MINLP).
It refers to problems with integer and non-integer variables and non-linear functions in the objective function and/or the constraints. Several works such as [22,84,147,156] have used MINLP to present their optimization problems.

Markov Decision Process (MDP).
It provides a mathematical framework to model and analyzes problems with stochastic and dynamic systems. Several works have used MDP to model scheduling problem in Fog computing such as [9,39,115,134].

Other.
There are also some other optimization modeling approaches in the literature of scheduling applications in Fog computing such as game theory [49], lyapunov [19,100], and mixed integer programming [6,116].

QoS Constraints
The formulated optimization problem usually contains several constraints, incurring higher complexity compared to unconstrained problems. In this work, we classify techniques based on the QoSrelated constraints applied to the main formulated problem into 1) deadline such as [32,128,140], 2) energy such as [11,146], and 3) hybrid (i.e., any combination of deadline, energy, and cost) such as [22,46,100].

Discussion
In this section, we discuss the effects of identified optimization characteristics' elements on the decision engine and describe the lessons that we have learned. Besides, we identify several research gaps accordingly. Table 4 provides a summary of characteristics related to optimization problems in Fog computing.

5.6.1
Effects on the decision engine. The optimization characteristics affect the decision engine in various aspects, as briefly described below.
1. Objective number and parameters: Simultaneous optimization of multi-objective problems usually incur higher complexity for the decision engine. Also, when the number of key parameters in a multi-objective scheduling problem increases, finding the best parameters' coefficients becomes a critical yet challenging step.

2.
Problem modeling: It can affect the choice of placement technique as some specific algorithms and techniques can be used to solve the scheduling problem. For example, several traditional LP and ILP tools and libraries exist to solve LP and ILP scheduling problems.
3. QoS constraints: They incur hard or soft constraints and limitations on the main objective/objectives of the scheduling problem, which intensify the complexity of the scheduling problem. The decision engine should satisfy these constraints either using classic Constraint Satisfaction Problem (CSP) techniques or using customized approaches.

Lessons learned.
Our findings regarding the optimization characteristics in the surveyed works are briefly described in what follows: 1. The main perspective of optimization for almost 75% of works is IoT, while for the rest of the works is hybrid and system by 15% and 10%, respectively. The main perspective element affects how some metrics are evaluated. For example, when evaluating the energy consumption in the IoT perspective, the energy consumed by the surrogate servers for the execution of tasks is overlooked. However, in the system and hybrid perspectives, the energy consumption of all resource providers and all entities in the systems are evaluated, respectively.

2.
Considering objective numbers in the optimization problem, the works are almost equally divided into single and multiple objective numbers. Overall, the majority of works studied time and/or energy as their main optimization metrics. While the works with an IoT perspective follow the same trend for the optimization metrics, the proposals with a system perspective almost consider the cost as their main optimization parameter. Also, the hybrid perspective proposals often consider a combination of time, energy, and/or cost as their main optimization metrics.
3. In problem modeling, the greater number of works have used either MDP or MINLP (each with roughly 25% of proposals) to formulate their problem. Also, some works initially had modeled their work as MINLP and then defined the MDP accordingly, such as [22,56]. The rest of the works have used MILP (almost 15%), ILP (almost 15 %), and other optimization modeling approaches.

4.
Almost 25% of works defined single or multiple QoS constraints for their problem, among which 90% have considered a single constraint, and the rest went for two QoS constraints. Among the QoS constraints, the deadline by 90% is the most used constraint in all works.

Research Gaps.
We have identified several open issues for further investigation that are discussed below: 1. The main part of works in the literature either consider optimization problems from IoT devices/users. However, only a few works have considered IoT and system perspectives simultaneously (i.e., hybrid). Optimizing either of these perspectives can negatively affect other perspectives. To illustrate, when the principal target is minimizing the energy consumption of IoT devices, the majority of components or tasks are placed at FSs or CSs. However, it may negatively affect the energy consumption of resource providers and even increase the aggregated energy consumption in the environment. Hence, further investigation on hybrid optimization perspectives and mutual effects of different perspectives is required.

2.
The cooperation among the resource providers (i.e., FSs, CSs) is an essential factor in offering higher-quality services. Proposals in the system and hybrid perspectives can also consider other metrics such as trust and privacy index for resource providers and study how they affect the overall performance.
3. QoS constraints are set to guarantee a minimum service level for end-users. In current literature, most of proposals have focused on the deadline as the constraint. However, several other parameters such as privacy, security, and monetary cost and their combination as hybrid QoS constraints are not studied.

DECISION ENGINE CHARACTERISTICS
The requirements of IoT applications in Fog computing can be satisfied if incoming IoT requests can be accurately scheduled based on the characteristics of application structure, environmental architecture, and optimization problems by the decision engine. The main responsibilities of the decision engine are organizing received IoT requests and solving the optimization problem through a placement decision while considering contextual information. Fig. 7 presents the main elements in decision engine, as described in what follows:

Deployment Layer
Decision engines can be deployed on servers at different layers unless the servers do not have sufficient resources to host them. Based on the deployment layer of the decision engine, the current literature can be classified into:

Admission Control
The admission control presents the behavior of decision engines when new requests arrive. It denotes how the new requests are queued and organized by the dispatching module for placement.
6.2.1 Queuing. Decision engine may use different queuing policies when incoming IoT requests arrives. Based on queuing policy, we classify works into 1) First-in-First-Out (FIFO) such as [32,39,125] and 2) Priority-based where incoming requests are sorted based on their priority (e.g., deadline) [45,50,128].

Dispatching
Mode. The dispatching module forwards requests from input queue to the placement module. Based on the selection policy of dispatching module, current literature can be classified to 1) single model where only one task at a time is dispatched for placement [22,47,143] and 2) batch model where a set of tasks are forwarded to placement module [4,40,98].

Placement Technique
Placement technique is the actual algorithm used to solve the optimization problem. Each placement algorithm has its advantages and disadvantages. Hence, it should be carefully selected based on the properties and dynamics of applications, users, environment, and deployment layer. We classify placement techniques based on their approach to find the solution into two broad categories: 6.3.1 Traditional. In this approach, the programmer/designer defines the required logic of policies for the placement technique. The traditional placement technique can be further divided into three subcategories: 1. Direct Optimization: In this category, the optimization problem will be solved using classical optimization tools either using 1) Exact approach to find the optimal solution such as [18,25] or 2) Approximation approach to find a near optimal solution such as [11,84,136].

2.
Heuristics: These algorithms are a set of typically problem-dependent algorithmic steps to find a feasible solution for the problem. Heuristics usually scale well as their TC is low while they do not guarantee finding the optimal solution of the problem [13,14,28,70].
3. Meta-heuristics: Meta-heuristics are composed of several advanced heuristics and typically are problem-independent, such as Genetic Algorithm (GA) and Simulated Annealing (SA). Although these algorithms usually perform better than heuristics, similarly they cannot guarantee to find the optimal solution. Several works such as [33,40,90,132] used meta-heuristics.

Machine Learning (ML).
ML is a family of algorithms that can learn the required policies for placement techniques from historic data. ML algorithms scale reasonably well, however, they require accurate and ideally large samples of historic data. The ML-based placement techniques can be further divided into three subcategories: 1. Supervised Learning: These algorithms learn by using labeled data as its input. Type of problems are regression and classification, and some of the algorithms are linear regression and logistic regression. Some works in the current literature used supervised ML for the placement technique such as [57,146,147].

2.
Unsupervised Learning: These algorithms are trained using unlabelled data, contrary to supervised ML, without any guidance. Some unsupervised algorithms are K-Means and fuzzy C-Means. Some works in the current literature used unsupervised ML for the placement technique such as [117,120,143].

Reinforcement Learning (RL):
In these algorithms, agent/agents learn the required policy for placement technique by interaction with an uncertain and potentially complex environment. It does not require pre-defined data, and type of problems are exploitation or exploration. The current RL-based literature in scheduling IoT applications can be divided into 1) Multi-Armed Bandit (MAB) which are among the simplest RL problems such as [127,156], 2) Deep RL (DRL) where deep learning is used in RL such as [9,89,125], and 3) Distributed DRL where several agents works collaboratively in a distributed manner for efficient learning such as [22,39,129].

Advanced Features
To fully utilize the potential of the Fog computing paradigm, several advanced features can be augmented with decision engines to capture high dynamics of this paradigm, described below: 6.4.1 Mobility Support. A significant number of IoT devices are moving entities (e.g., vehicles), requiring connected service through their path. So, decision engines should manage the migration process of application components and find suitable surrogate servers accordingly. Several works such as [11,38,90,138] address mobility and migration management challenges alongside scheduling IoT applications.

Failure Recovery.
In highly dynamic systems such as Fog computing, failure may happen due to software or hardware-related issues. So, application components faced with failure should be re-executed. Some works consider failure recovery mechanisms in their decision engines such as [16,38,40].

High Scalability.
As a large number of IoT devices and servers exist in the Fog environment, mechanisms and algorithms used in the decision engine should be highly scalable and provide well-suited performance when the system size grows. Several works have studied the scalability feature of their techniques when the number of IoT applications and servers increases or discussed how their distributed techniques work efficiently in large systems such as [30,53,73].

High
Adaptability. This feature ensures that the decision engine dynamically captures the contextual information (i.e., application, environment, etc), and updates the policies of placement techniques accordingly. In Fog literature, several works such as [31,39,108,153] offer solutions with high adaptability.

Implementation
The implementation characteristics of decision engines are studied based on the following criteria: 6.5.1 Language. Different programming languages are used for the implementation of decision engines, while the majority have used Python [39,74], Java [40,119], and C++ [12,124].
6.5.2 Source Code. Open-source decision engines help researchers to understand the detailed implementation specifications of each work, and minimize the reproducibility effort of decision engines, especially for comparison purposes. Some works such as [24,110,134] have provided the source code repository of their decision engines.

Time Complexity (TC)
. TC of each placement technique presents the required time to solve the optimization problem in the worst-case scenario. It directly affect the service startup time and the decision overhead of each technique. Based on the current literature, we classify the TC into 1) Low the solution of optimization problem can be obtained in polynomial time where the maximum power of variable is equal or less than two (i.e., ( 2 )) [9,16,39], 2) medium where the time complexity is polynomial and the maximum power is less than or equal to 3 (i.e., ( 3 )) [13,70,104], and 3) High for exponential TC and polynomials with high maximum power [18,25,128].

Discussion
In this section, we describe the lessons that we have learned regarding identified elements in decision engine characteristics of the current literature. Besides, we identify several research gaps accordingly. Table 5 provides a summary of decision engines-related characteristics in Fog computing.
6.6.1 Lessons learned. Our findings regarding the decision engine characteristics in the surveyed works are briefly described in what follows: 1. Almost 85% of surveyed works deployed the decision engine at the Edge layer in the proximity of IoT devices. Since the Edge servers can be accessed with lower latency and higher access bandwidth, deployment of decision engines at the Edge can reduce the startup of IoT applications. However, the Edge devices should have sufficient resources to run the decision engine. Some proposals (about 10%) also deployed the decision engine on IoT devices. Deployment of a decision engine on IoT devices provides more control for IoT devices, especially mobile ones. It eliminates the extra overhead of communication with surrogate servers for making a decision. However, IoT devices often have very limited resources that are incapable of running powerful decision engines.

2.
The queuing is an important element in the admission control that almost 80% of the works have not studied. Since most of works have considered several IoT devices in the environment, several IoT requests may arrive in each decision time-slot with a high probability. Hence, different queuing models can dramatically affect the decision engine performance and the QoS of endusers. FIFO and priority queue share the same proportion of proposals among the works that mentioned their queuing policy. Also, in priority-based queuing, almost all works have considered the deadline of applications or tasks as their main priority metric. Moreover, for the policy of dispatching module, about 75% of works selected single dispatching while 25% of works studied batch dispatching policy. Since different IoT requests may arrive in the same decision time-slot, the batch dispatching policy helps study the mutual effects of IoT applications with diverse resource requirements in the placement decision.
3. The traditional placement techniques are used in almost 60% of the proposals, while the ML-based placement techniques are studied in the rest of the works (almost 40%). However, the number of ML-based placement techniques has significantly increased in recent years. In traditional placement techniques, direct optimization, heuristics, and meta-heuristics share the same proportion of proposals. Also, in meta-heuristics techniques, the majority of works used population-based meta-heuristics, especially different variations of the GA. In the ML-based techniques, the majority of proposals have used RL-based techniques (almost 70%), specially DRL. Moreover, in the DRL techniques, the larger number of works used centralized DRL techniques such as DQN. However, the exploration and convergence rate of centralized DRL techniques are very slow. Thus, several studies have recently been conducted to adapt distributed DRL (i.e., DDRL) techniques for resource management in Edge/Fog computing environments, such as [39,53,73], to improve the exploration cost and convergence rate of the DRL techniques.

4.
In advanced features, almost 25% of proposals embedded different mechanisms (i.e., traditional or ML-based) for the mobility management of IoT devices and migration of applications' constituent parts. Also, about 25% of studied works, mostly ML-based techniques, offer high adaptability features in their decision engine. However, traditional works often neglect to provide different mechanisms to support high adaptability. This is mainly because the scheduling policies are not statically defined in ML-based techniques. Hence, as the environmental or application properties change, the policies can be learned and updated accordingly. However, in the traditional scheduling techniques, updating the scheduling policies according to dynamic changes in environmental or application properties is very costly and time-consuming. Almost 20% of proposals studied different mechanisms to support high scalability feature, either using ML-based techniques or traditional approaches. In advanced features, the failure recovery mechanisms and techniques in scheduling are not well-studied and only a few works embedded these mechanisms in their scheduling techniques.

5.
Considering the implementation of the techniques, almost 50% of the works mentioned their employed programming language. Java and python programming language are the most-employed programming language and are almost equally used in different proposals. However, Python is mainly used for ML-based techniques and direct optimization techniques, while Java is mostly used to implement traditional decision engines. Moreover, only about 10% of proposals shared their open-source repositories with researchers and developers among the surveyed works. Finally, about 65% of proposals discussed the TC of their works, among which almost 80% proposed decision engines with low TC while some proposals (almost 10%) went for medium TC and few works (almost 10%) proposed decision engines with high TC. The high TC proposals are among the direct optimization category of traditional approaches. While these high TC proposals cannot be currently adapted to large-scale Edge and Fog computing environments, they can find the optimal solution in small-scale problems. Hence, they can be used as a reference for the evaluation of other proposals. 6.6.2 Research Gaps. We have identified several open issues for further investigation that are discussed below: 1. The admission control concept in terms of different queuing, dispatching, and their mutual effect is not well studied in the current literature. Also, the greater number of works consider a single task dispatching model and overlook batch placement of applications, especially for applications with dependent tasks.

2.
While traditional placement techniques (e.g., heuristics, meta-heuristics) are studied well in the literature, ML-based techniques are still in their infancy. Due to the lack of a large number of datasets, supervised and unsupervised ML have not been thoroughly considered. Also, the majority of employed RL techniques are centralized approaches, neglecting collaborative learning of multiple distributed agents for better efficiency and lower exploration costs.
3. Although all servers and devices are prone to failures, among advanced features, failure recovery mechanisms, algorithms, and their integration with the placement technique is the leaststudied concept. Even the best placement techniques cannot complete their process in real-world scenarios unless a suitable failure recovery mechanism is embedded.

4.
In the surveyed works, there is no proposal to study all the four identified elements in the advanced features (i.e., mobility, failure recovery, scalability, and adaptability) and describe the behavior and mutual effects of these elements on each other and decision engine.

5.
Among the studied literature, none of the works has studied the privacy problem from different perspectives, such as end-users' data privacy, resource providers' privacy, and the decision engine's mechanisms for improving privacy.

PERFORMANCE EVALUATION
Different approaches and metrics have been used by the research community to evaluate the performance of their techniques. Identifying and studying these parameters helps to select the best approach and metrics for the implementation of new proposals and fair comparisons with other techniques in the literature. Fig. 8 presents a taxonomy and the main elements of performance evaluation, described below:

Approaches
The performance evaluation approaches can be divided into four categories, namely analytical, simulation, practical, and hybrid. There are different important aspects to consider when selecting an approach for the evaluation of proposals, such as credibility, implementation time, monetary cost, reproducibility time, and scalability. Fig. 9 presents a qualitative comparison of different approaches used in performance evaluation. 7.1.1 Analytical. One of the popular approaches for the evaluation of different proposals is analytical tools. Usually, the implementation time, reproducibility time, and monetary cost of analytical tools are low, and scalable experiments can be executed. However, the credibility of such experiments is low since the dynamics of resources, applications, and environment cannot be fully captured and tested. Matlab is among the most popular tools that is either used directly [28,147,147] or integrated with some other libraries such as Sedumi 1 [116]. Also, C++ based analytical tools have been used in the literature, such as [12,84].    [43,77] is among the most popular simulators for Fog computing [40,64,72,129]. Besides, several researchers have used Cloudsim [17] such as [25,145] or SimPy 2 such as [31,94] to simulate their scenarios in Fog computing.

Practical.
The most credible approach for the evaluation of proposals is practical implementation. However, due to high monetary cost, implementation time, and reproducibility time, it is not the most efficient approach for different scenarios, especially evaluations requiring high scalability.
In the literature, few works such as [11,32,117,119] evaluated their proposals using small-scale practical implementations.

Hybrid.
In this approach, researchers evaluate their proposals using practical implementations in small-scale and simulators or analytical tools in large-scale. Although implementation and reproducibility time of this approach is high, it provides high scalability and credibility. In the literature, few works such as [39,74,110] follow the hybrid approach.

Metrics
The metrics used in performance evaluation in Fog computing are directly or indirectly related to the optimization parameters and system properties. Based on the nature and popularity of these metrics in the literature, we categorize them into 1) time (e.g., deadline, response time, execution time, makespan) [24,25,70,89], 2) energy (e.g., battery percentage, saved energy) [13,20,55,55], 3) monetary cost (e.g., service cost, switching cost) [9,90,94,100], and 4) other metrics (e.g., number of interrupted tasks, resource utilization, throughput, deadline miss ratio) [14,16,38,49]. Also, we consider 5) decision overhead as an important evaluation metric to study the overhead of proposals (often in terms of time and energy), used in some works such as [39,64,102,142].

Discussion
In this section, we describe the lessons that we have learned regarding identified elements in the performance evaluation of the current literature. Besides, we identify several research gaps accordingly. Table 6 provides a summary of characteristics related to performance evaluation in Fog computing.

Lessons learned.
Our findings regarding the performance evaluation in the surveyed works are briefly described in what follows: 1. More than half of the works used the simulation as their performance evaluation approach while 30% of the proposals used an analytical approach. The practical and hybrid approaches equally share the rest of 20% of the works. For the analytical approach, the most of works used Matlab or Python programming languages, while Java and Python are mostly used for the simulation approach. In practical and hybrid approaches, Java and Python are equally employed in proposals.

2.
As the performance evaluation metric, time and its variations (e.g., response time, makespan) are used in more than 80% of the works. The second-highest-used metric is energy at 35%. However, the decision overhead and cost are only studied in 15% of the works. Besides, less than 5% of the proposals studied the performance of their scheduling technique using all the identified metrics.

Research Gaps.
We have identified several open issues for further investigation that are discussed below: 1. Although the monetary costs of sensors and edge devices (e.g., Rpi, Jetson Platform) have reduced and they are highly available in different configurations, compared to a few years ago, the majority of proposals still consider analytical tools and simulators as their only approach for performance evaluation. While some works have considered practical and hybrid approaches for the performance evaluation of their work, further efforts are required to study the dynamics of the system, resource contention, and collaborative execution of the application in real environments, especially considering new machine learning techniques such as DRL and DDRL [39,110].

2.
The decision overhead of proposals has direct effects on users and resources in terms of the startup time of requested services and resource utilization. To illustrate, not only do healthcare applications require low response time, but they also need low startup time, especially for critical applications such as emergency-related applications (e.g., heart-attack prediction and detection). Also, the overhead of proposals can severely affect the resource usage and energy consumption of servers, especially battery-constrained ones. Among the techniques considered decision overhead as a metric, they mostly focus on time while other metrics (e.g., energy, cost) need further investigation.

SCHEDULING TECHNIQUE: IMPORTANT DESIGN OPTIONS
In this section, we discuss the real-world characteristics of application structure and environmental architecture and accordingly present several guidelines for designing a scheduling technique.
1. The number of IoT applications is constantly increasing in different application domains. The majority of these applications are defined as a set of dependent modules/services [155]. Besides, sharing and reusing modules/services for faster development and better management of applications is of paramount importance. Moreover, dependent IoT applications are usually modeled as a graph of tasks and their respective invocations. In this case, IoT applications with monolithic and independent design can also be defined as an application graph with only one module and an application graph with several modules where the size of invocations is zero, respectively. Hence, we consider IoT applications with dependent modules/services (i.e., modular and loosely-coupled categories) as the main architectural design choices in the application structure. Accordingly, the decision engine requires a component for identifying and satisfying the constraint among modules/services.

2.
Besides, in a real-world scenario, application modules have different characteristics (e.g., computation size, input size, ram usage). Thus, the best assumption for application modules is applications with heterogeneous granularity specifications. As the number of contributing parameters and the dynamicity of the application elements increases, capturing the application parameters with temporal patterns for efficient scheduling decisions becomes more complex [39,129]. Although traditional-based placement techniques (e.g., heuristic, meta-heuristic) often work well in general scenarios, they fail to adapt to continuous changes and dynamic contexts. ML-based decision engines, such as RL, can more efficiently work in a dynamic context and provide higher adaptability.

3.
In large-scale Fog computing environments, numerous IoT applications with different workload models and hybrid CCR may exist. Hence, the decision engine requires an admission control component with an appropriate queuing mechanism (based on application requirements) to manage diverse incoming requests and prioritize them for making the decision.

4.
Regarding the environmental architecture, the most generalized scenario is when the environment consists of several heterogeneous IoT devices, several heterogeneous FSs, and multiple heterogeneous CSs. Also, the required mechanisms for intra-tier and inter-tier cooperation among servers should be embedded to support diverse IoT application scenarios, such as mobility. Besides, multiple distributed servers can collaboratively provide better performance for the execution of IoT applications. Moreover, different fault domains can be prepared to improve the availability of services. However, as the number of IoT applications and available servers in the environment increase, the complexity of making decisions increases. Hence, the optimal scheduling decision cannot be obtained in a timely manner. Consequently, other placement techniques such as heuristics and ML-based techniques should be employed to obtain the scheduling decision in a reasonable time.

5.
The decision engine can be implemented as a set of distributed services/microservices. A decision engine developed as a monolithic application may not be able to be deployed on a single server, especially on resource-limited FSs. Hence, distributed deployment of decision engine components on several distributed servers can provide several benefits: 1) more efficient deployment of resource-limited devices, 2) provides better fault tolerance 3) offers better scalability 4) support different deployment models (e.g., deployment of decision engine on FSs, CSs, or hybrid on both FSs and CSs). Hybrid deployment of decision engine components on both FSs and CSs can lead to a better user experience for end-users. To illustrate, applications requiring low latency and startup time can be managed at the low-level FSs (i.e., at the Edge), and then be scheduled based on the decision engine deployed at the Edge. However, application requests that are insensitive to latency or startup time can be forwarded to CSs for scheduling.
6. Regardless of application and environmental characteristics, failure recovery mechanisms and policies should be integrated into any decision engine. Independent failures and the nondeterministic nature of any components (either hardware or software) in distributed systems cause the most impactful issues in distributed systems. If the decision engine, which manages the scheduling and execution of incoming IoT application requests, does not have an appropriate failure recovery mechanism, the smooth execution of the whole system stalls.

FUTURE RESEARCH DIRECTIONS
This section presents future research directions, guiding researchers to further progress in the field of Fog computing.
Microservices-based applications. The popularity of microservices for the deployment of IoT applications is due to their loosely-coupled design, modularity, and the capability of microservices to be shared among multiple IoT applications. But, it may incur data consistency and data privacy challenges. To overcome these challenges, the placement techniques should consider the context of applications and data before sharing microservices.
Practical Container orchestration in Fog computing. Orchestrating container-based IoT applications is well studied in the cloud computing paradigm. However, in Fog computing, in which CSs and FSs collaborate to run an application, several deployment models of orchestration techniques are available. To illustrate, the master node can either be deployed on a FS or CS. When the master node runs on a FS, the communication overhead and latency for end-users will be reduced. However, the master node will use the most of resources on the FS for the cluster management, especially for resource-limited FSs. Also, when the master runs on a CS, the startup time and application latency will be negatively affected. Thus, based on the application structure and its goal, different container orchestration models should be studied to find the best deployment model according the application scenario. Several practical studies can be conducted to find which deployment model is suitable for each IoT application scenario in terms of communication overhead, the startup time of services, memory footprint, failure management, load balancing, and scheduling.
Hybrid scheduling decision engines. Usually, decision engines only use one placement technique for different IoT applications. However, the requirements of IoT applications are heterogeneous, where one application is sensitive to startup time while the extremely high accuracy is not important, or vice versa. Besides, decision engines should be adapted to work with either single or batch placement approaches. Hence, context-aware decision engines with a suite of placement techniques can be implemented to address the requirements of different IoT applications.
Systems for ML. Due to advancements in ML techniques and their rapid adoptions across many IoT applications, it creates new demand for specialized hardware resources and software frameworks (e.g., Nvidia GPU-powered Jetson, Google Coral Edge Tensor Processing Unit (Edge TPU)) for Fog computing. New systems and software frameworks should be built to support the massive computational requirement of these AI workloads. Besides, these systems can be a potential target for the deployment of decision engines due to their high computational capacity.
ML for systems. While ML systems themselves are becoming mature and adopted into many critical application domains, it is equally important to use these ML techniques to design and operate large-scale systems. Adopting the ML techniques to solve different resource management problems in Edge/Fog and Cloud is crucial to managing these complex infrastructures and workloads. Moreover, majority of ML techniques are not optimized to run on resource-constrained devices. To illustrate, consider an efficient ML model trained for resource management. Many resourceconstrained devices require full integer quantization to run the trained model. However, post quantization of trained models is not always possible and in some cases they cannot be efficiently converted. As a result, a study on requirements for the efficient execution of resource management ML models on resource-limited FSs should be conducted.
Thermal management. The temperature of FSs (e.g., racks of Rpi or Nvidia Jetson platform), especially those executing large workloads, increases significantly. So, the cooling systems should be embedded to avoid system breakdown. Hence, a study on the temperature of these devices based on their main processing and communication modules can be conducted to find the respective temperature dynamics in different application scenarios and workloads. Moreover, lightweight thermal management software systems for FSs can be designed to control the temperature dynamics of devices. Also, the thermal index can be added as an important optimization/decision parameter alongside other currently available parameters (e.g., time, energy, cost) for the placement techniques.
Trade-off between execution cost of IoT devices and resource providers. The goal of scheduling algorithms is to minimize the execution cost of applications either from IoT or resource providers' perspectives. However, some parameters such as energy consumption or carbon footprint should be considered from both perspectives. Hence, not only is minimizing these parameters from either perspective critical to reducing total energy consumption, but a trade-off parameter between the execution cost of IoT devices and resource providers can be designed, aiming at total energy or carbon footprint minimization.
Privacy aware and adaptive decision engines. Data-driven and distributed scheduling approaches are gaining popularity due to their high adaptability and scalability. However, sharing raw data of users or systems incurs privacy issues. To illustrate, in DDRL-based scheduling techniques, sharing experiences of multiple agents significantly reduce the exploration costs and improve convergence time of DDRL agents while incurring privacy concerns when raw agents' experiences are shared. Accordingly, privacy-aware mechanisms for sharing such data (e.g., agents' experiences) can be integrated with these highly adaptive distributed scheduling techniques.
In this paper, we mainly focused on scheduling IoT applications in Fog computing environments. We identified several main perspectives that play an important roles in scheduling IoT applications, namely application structure, environmental architecture, optimization characteristics, decision engines properties, and performance evaluation. Next, we separately identified and discussed the main elements of each perspective and provided a taxonomy and research gaps in the recent literature. Finally, we highlighted several future research directions for further improvement of Fog computing.