{'fraction_non_alphanumeric': 0.034840457799166905, 'fraction_numerical': 0.011546083228859142, 'mean_word_length': 5.2672665061462425, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Decentralized Socio-technical Services and Applications for the Internet of Things -- A Testbed Self-Integration

The Internet of Things comes along with new challenges for experimenting, testing, and operating decentralized socio-technical systems at large-scale. In such systems autonomous agents interact locally with their users and remotely with other agents, to make intelligent collective choices, via which they self-regulate the consumption and production of distributed resources. For instance, self-management of traffic flows in Smart Cities. While such complex systems are usually deployed and operated using centralized computing infrastructures, their socio-technical nature requires new value-oriented design paradigms empowering trust, transparency, and alignment with citizens' social values, such as privacy preservation, autonomy, and fairness among citizens' choices. Currently, instruments and tools to study such systems and guide the prototyping process from simulation, to live deployment, and ultimately to a robust operation, are missing or not practical. This paper bridges this gap by introducing a novel testbed architecture for decentralized socio-technical systems running on IoT. This architecture empowers the seamless reusability of (i) application-independent decentralized services by an IoT application, and (ii) different IoT applications by the same decentralized service. This dual self-integration promises IoT applications that are simpler to prototype and can interoperate with services during runtime to self-integrate more complex functionality. Such self-integration provides stronger validation and improves efficiency of resource utilization: computational resources are shared, cutting down deployment and operational costs. Pressure/crash tests during continuous operations of several weeks, with more than 3000 agents joining/leaving, 150,000 parameter changes, and 3 million communicated messages/day, confirm the robustness and practicality of the testbed architecture.


Introduction
The Internet of Things (IoT) manifests a predominant role on how complex socio-technical systems are designed, operated and managed.
Smart Cities turn into organic ecosystems of ubiquitous sensors, autonomous vehicles, and personal pervasive devices that are massively interconnected and distributed [1,2,3]. New opportunities arise to control and manage socio-technical systems in real-time as the means to cope with uncertainties and continuous change: Self-improving socio-technical operations by seamlessly self-integrating decentralized services that measure, learn, optimize, and adapt [4,5,6], i.e. load-balancing transport or power networks to prevent traffic congestion and blackouts, respectively. However, the IoT complexity, heterogeneity, scale, infrastructural cost, and privacy concerns have so far application agent and the service agent. In practice, these two agents can run on the same computational node, i.e. on a user's device such as a smart phone, or at two remote nodes, i.e. on a user's device, a cloud node, or a crowdsourced community server 1 .
Prototyping IoT applications and services to support, in particular, multiple self-integration scenarios, requires testing and refinements at multiple stages that start from simulations, move to live deployments, and ultimately to high Technical Readiness Level (TRL) 2 operations. Maintaining different implementations, or changing the code back and forth to validate new functionality is costly and complex [10,3]. Experience shows that such flexibility in existing toolkits for decentralized multi-agent systems is extremely scarce [1,2]. This paper shows that in practice they could not serve the self-integration scenarios envisioned in this paper. This barrier is overcome by introducing a prototyping toolkit that extends and improves earlier work [11]: The Distributed and Intelligent Social Computing (DISC) toolkit. DISC toolkit provides support for IoT devices, i.e. software agent running on smart phones, a new efficient networking module, a new scalable logging infrastructure for system monitoring and analysis, as well as improved design to limit earlier severe memory leaks and synchronization problems.
The testbed architecture with the two software agents is realized and experimentally evaluated with real-world data, pressure/crash tests, and long-lasting operations of several weeks. The complex operations of two decentralized socio-technical services are integrated in DISC as a proof of concept: (i) I-EPOS, the Iterative Economic Planning and Optimized Sections [5], and (ii) DIAS, the Dynamic Intelligent Aggregation Service [6]. I-EPOS 3 performs decentralized combinatorial optimization using learning agents with structured interactions. In contrast, DIAS 4 performs real-time collective measurements over a dynamic unstructured network of agents, i.e. a network whose agents can arbitrary join, leave or fail, while their input data continuously change. Both services empower highly sophisticated IoT application scenarios such as traffic flow optimization, power peak-shaving, load-balancing of bike sharing stations, participatory crowd-sensing of mobility, traffic, environment and others [12,13,5,6]. Results confirm the self-integration capability, and the performance benchmarks validate the robustness of the testbed architecture in scenarios of continuous change and adaptation, with more than 3000 agent joining/leaving, 150,000 parameter changes, and 3 million communicated messages per day. These results provide new insights to communities, government bodies, system operators and utilities on how to manage, operate and In summary, the contributions of this paper are outlined as follows: (i) A conceptual testbed architecture that facilitates dual self-integration of various decentralized services by an IoT application, and different IoT applications by a decentralized service. (ii) The realization of the conceptual testbed architecture by abstracting the software engineering complexity in two software agents and their interactions in a generic communication protocol. (iii) An improved and extended distributed prototyping toolkit for decentralized socio-technical systems of TRL 6 running on IoT. (iv) Improvements of the I-EPOS software artifact [5] that moves from simulations to live deployments and a demonstrated TRL-6 continuous operation. (v) A proof of concept based on the self-integration and experimental evaluation of two decentralized services for IoT applications under highly dynamic environments.
The rest of this paper is outlined as follows: Section 2 reviews relevant previous work. Section 3 introduces the testbed architecture, and the realization protocol. Section 4 illustrates the DISC toolkit, and Section 5 introduces the two studied services. Sections 6 and 7 illustrate the experimental methodology, and evaluations, respectively. Finally, Section 8 concludes this paper and outlines future work.

Related Work
Previous research in the field of experimental IoT has been extensively reviewed and compared [1,2,7,32,33,34]. Such research focuses on different aspects and contributes various instruments and tools. Physical testbeds equip researchers with deployed and ready-to-use physical devices, simplifying the design and evaluation of novel IoT systems and services (e.g., network protocols, Big Data algorithms, city-wide IoT services) under realistic operational conditions [14,15,16,17,18,19]. FIT IoT-Lab [14] is one example, as it provides a federation of testbed platforms with 2728 sensors, some of which can be reprogrammed to test novel protocols. Another example is the SmartSantander [15] testbed with a city-wide scale (∼20,000 sensors). SmartSantander nodes act only as data sources and can be configured (centrally via a management plane) to implement applications such as environmental monitoring. However, in physical testbeds the domain/project-specific requirements often determine the design and technological aspects (i.e., communication protocols) of sensors, smart object and middleware. This limits their reusability in different domains and applications.
To tackle such challenges, the PaaS (platform-as-a-service) model has been studied and utilized. The PaaS model leverages standard interfaces and interoperability measures, to provide researchers with tools to rapidly develop, execute, and manage IoT systems without the complexity of building and maintaining the infrastructure [3]. This enables the design and The abstraction indicates the three possible levels of applied abstraction: H: Hardware abstraction by providing software routines to access the hardware via programming interfaces. D: Device abstraction by having virtualized counterparts for each IoT device at the system-level. The AO indicates whether virtual counterpart is an agent. Agents are networked software components that autonomously perform specific tasks on device/user behalf by interacting with other agents and with their environment [10] and S: Indicates service-level abstraction by providing common communication protocols for IoT services. Data locality, refers to local processing of data, while autonomy is the ability of the device to autonomously interact and actuate its function. decentralized control indicate the existence/lack of central control entities at the service-level. Reusability refers to the ability to reuse the Hardwares (H), Devices (D), or Services (S) in different application and scenarios. The interoperability illustrates the utilized communication paradigm. Technical refers to technological approaches (e.g., bluetooth), syntactical the shared message formats, and semantical the use of shared ontologies and knowledge representation [32]. deployment of cross-application IoT platforms [3]. Xively 5 is an example of such platform in the context of distributed cloud-based applications with a centralized control plane, where different tasks are executed in separate platforms and devices. For instance, the application-level functions can be executed in different virtual and real entities to reduce latency and bottlenecks. However, PaaS approaches often neglect socio-technical requirements, such as data locality, privacy, autonomy, and decentralized control.
Agent-based computing has been used extensively to enable cooperative, decentralized, dynamic, and open IoT systems [32]. In agent-based systems, agents autonomously interact and cooperate based on (typically) asynchronous message passing mechanisms to perform a task or a service. Shared communication standards facilitate agent interoperability and allows for incorporating heterogeneous resources. Lysis [3] introduces a PaaS model with virtualized autonomous social agents, allowing for the deployment of fully distributed applications. ACOSO-Meth [10] introduces an agent-oriented architecture based on IoT smart objects. It also provides a taxonomy for assessing system-level requirements and technological readiness of IoT systems. AoT (Agents of things) [23] studies an agent-based vision of IoT system, where each device/agent is infused with intelligence; Meaning it is augmented with software components allowing it to reason, negotiate, and delegate. iCore [30] introduces a cognitive management framework where cognition and proximity are used to handle device/service volatility in Smart City environments. The SIoT framework [24] leverages synergies between IoT and social networks to identify social relations between IoT objects, and integrate IoT devices into social networks. While agent-based approaches utilize device virtualization and address some socio-technical considerations, on the service-level they often suffer from lack of standard interfaces and interoperability. Thus, to reuse a specific service in different applications, the code, and communication protocol should change. This paper utilizes service abstraction to enable reusability of devices and services in various application domains.
Finally, IoT systems are operated in increasingly dynamic and complex environments [31], where during system runtime, devices can fail, users might join/leave, communication across devices and agents become disrupted, system goals and requirement can vary, and new services are required. Not all such changes can be foreseen during the initial design phase. Often it is infeasible for a centralized controller to have knowledge of all such changes in a timely manner. Self-* approaches, and autonomic computing [35,36] have been proposed as means to enable systems to handle such changes at runtime, with minimal human intervention [37,38,39,36]. To this end, the proposed testbed architecture facilitates the rapid prototyping and experimentation of IoT services that can handle dynamic environments (with high TRL) as well as autonomously initialize and include various devices and services during runtime. Table 1 illustrates a non-exhaustive comparison between relevant previous research, to provide insight into the current state of experimental IoT testbeds 6 .

A Conceptual IoT Testbed Architecture
This section introduces a conceptual testbed architecture designed to enable seamless reusability and self-integration of (i) application-independent decentralized services by an IoT application, and (ii) different IoT applications by the same decentralized service. Figure 1 illustrates the conceptual testbed architecture, and two examples for self-integrating different decentralized services with different IoT applications.
The proposed architecture utilizes two levels of abstraction: (I) IoT application level, and (ii) decentralized service level. At the IoT application level, this abstraction creates application agents: a piece of software lying on each user's IoT device, acting as the middleware for the communication with the decentralized service. Users' IoT devices provide the sensing and actuation for the system, can be of different types (e.g., sensors, mobile phones), and geo-spatially distributed. At the decentralized service level, this abstraction creates service agents, as counterparts for each application agent. These agents have a one-to-one mapping to the underlying IoT devices, and interact and cooperate with other service agents to execute the service 7 . Each service agent has the following tasks: (i) Receiving and managing data from the corresponding application agent (IoT device). (ii) Executing the requested service by interacting and cooperating with other service agents. Finally, (iii) Providing the outcome of the service to the device (possibly in form of control commands), and the user. Service agents are deployed and managed by the service operator, which can be a third-party mediator in sensing-as-a-service scenarios [33], or a community in case of participatory sensing applications [40]. Examples of third-party mediator include companies such as Waze 8 , Uber 9 , and Swiss Mobility 10 , while environmental monitoring [41], and urban sensing [42] are examples of participatory sensing applications deployed and managed by service communities.
The utilized dual abstraction creates a decoupling between the internal operations of the IoT application with the more complex functionalities of the decentralized service, and determines common interfaces and protocols between the two agents. The first abstraction level (application to services) facilitates the inclusion of heterogeneous devices, and their reusability in different services and scenarios. The second abstraction level (service to applications) simplifies the reusability and extension of decentralized services to new application domains, as the interface, communication logic, and protocols remain unaffected by changes in the IoT devices or the applications.
This architecture focuses on decentralized socio-technical IoT services with autonomous social agents, where there is no central authority to coordinate the agents and their actions. These agents interact locally with their users and remotely with each other to make intelligent collective choices, via which they can self-regulate the consumption and production of common resources. In this context, a service is essentially a distributed software running on multiple agents. Examples of such services include monitoring services [43], real-time analytics [6], planning and coordination systems [5], machine learning application [44], and distributed control systems [4]. The governance of the testbed depends on the application domain and the service. Smart City scenarios run by the municipality [45], Smart Grid [46], or smart supply chains [47] are examples cases where a central authority such as the municipality or the utility company governs the system. However, in participatory sensing scenarios, such as environmental monitoring [41], and urban sensing [42], the testbed can be self-governed by the users and the service community.

Communication Protocol & Runtime Cycle
To realize and operationalize the conceptual architecture, and enable the communication and self-integration between the two abstracted levels, a distributed protocol is designed. This generic communication protocols is application and service independent, hiding the processes of devices and agents, and fostering their interaction complexities. The protocol is outlined as follows: (i) The service operator initializes the service agents, and gateway. The gateway act as a bootstrapping proxy, adding online the service agents and connecting    The details of messages are illustrated in Section 3. This protocol treats the running services as blackboxes and create standard interface between different components of the testbed. Hence, different devices and services can be self-integrated at runtime. them to the corresponding application agent, The gateway is agnostic of data, the internal processes of the IoT application and the decentralized service 11 . (ii) It is assumed that IoT devices know the public address of the gateway. This is possible via the broadcastMsg:{GWAddr, servInfo} by the gateway, where GWAddr is the gateway address, and servInfo indicates the service. (iii) To connect to the decentralized IoT service, each application agent contacts the service gateway, registers itself, and the corresponding IoT device, via the regDevMsg:{devAddr, devInfo, servInfo}, where devAddr is the application agent address, devInfo determines the device type (e.g., sensor, mobile phone). (iv) In response, the gateway assigns a service agent to the IoT device informs the application agent via the asgnAgnMsg:{agnAddr}, where agnAddr is the address of the assigned service agent.
(v) The service operator submits a service request to the gateway, specifying its requested service, and execution metadata, via the servReqMsg:{servInfo, servMD}, where servMD the metadata required to execute the service, such as the number of service agents, number of devices, and their locations. (vi) The gateway receives the service request, and notifies the agents, via the readyMsg:{servInfo, servMD}. (vii) The service agents validates the service information, and its associated metadata and in turn notifies the gateway, via the agnReadyMsg:{agnAddr, servInfo}. (viii) When all agents are notified and ready to run the service, the gateway sends the execute service command, via the runServMsg:{servInfo}.
(ix) The service agent requests/receives data from the IoT device via the sensingMsg:{servInfo, data}, and submits control/actuation messages to the IoT device either periodically, ondemand, or at the end of the service execution via the actuationMsg:{servInfo, actuation}. (x) Finally, after the service is executed, the application agent, gateway, and the service operator are informed. Figures 2 illustrates the protocol sequence diagram, and runtime cycle.

Prototyping decentralized IoT Systems
There are several challenges in prototyping IoT applications and services: To ensure efficient and reliable performance, continuous testing and refinements are needed, from the early stages of simulation, to live deployment, to long-lasting stable operation. Additionally, maintaining different implementations or changing the code back and forth to validate new functionality or extending to new application domains is costly and complex. To address these challenges, and facilitate rapid design and testing of decentralized IoT services across different application domains, this paper introduces the Distributed and Intelligent Social Computing (DISC) toolkit. The DISC toolkit is based on an improved version of the general-purpose prototyping toolkit Protopeer [11], now made highly robust and efficient for long-term operation, Additional modules, all with a high Technical Readiness Level (TRL6) include: (i) software clients (application agents) for supporting of IoT devices, i.e. smart phones, (ii) an efficient networking module, and (iii) a scalable monitoring infrastructure, integrated to each computational node, for system monitoring and analysis.

Redesigned & Reengineered Protopeer
The Protopeer toolkit [11] is designed with the main goal of facilitating the rapid prototyping of P2P applications, and the transition from simulation to live environments. However, it shows several limitations in long-lasting high-pressure live deployments, such as memory leaks, lost messages, deadlocks, synchronization, and excessive thread counts. To address the above limitations, this paper introduces a redesigned and reengineered version of Protopeer for highly efficient and robust long-lasting experimentations. This new version have been tested extensively over several weeks under highly dynamic environments, with approximately 4000 node join/leaves, 150,000 runtime parameter changes, and over 3 million exchanged messages per day (Section 7.2). Figure 3 illustrates a summarized view of the internal architecture of the redesigned Protopeer. There are two core concepts within each Protopeer node: the peer, and the peerlet. The peer provides core functionality such as communication protocols (e.g., TCP messaging), and timers, and acts as the execution environment (container) for the peerlets. The peerlets, on the other hand, are independent modules that provide specific functionalities and tasks. Typically a node consists of a single Protopeer peer and multiple peerlets that collectively fulfill the required functionality of the service.
Two examples of peerlets are the communication topology peerlet, and the monitoring peerlet. The communication topology peerlet determines the service network topology and the communication logic, for instance, a tree-topology [48], or the gossip-based peer sampling for P2P systems [49] The monitoring peerlet stores and submits logs from different modules in the peer (e.g., peerlets) to the monitoring infrastructure (Section 4.3). By default, this peerlet includes three different logging modes: (i) Service Logger, which logs the specific  Figure 3: Internal architecture and modules of a computational node. The internal modules of each node is separated in two categories: deployment dependents, and deployment independent. This design has the advantage of allowing to first test a system in a controlled simulated environment, and gradually move to large-scale live deployment, as only the network level implementation (deployment dependent) should change.
application-level service logs, (ii) event Logger, which provided event-based logging, and insight into execution sequence, and (iii) memory Logger, which measures the total memory footprint of a peer in memory, including nested objects stored within containers 12 .

Communication Platform
This architecture utilizes a messaging protocol based on a fast and lightweight TCP/IP implementation using ZeroMQ for agent to agent communication. Each agent is instantiated with a single PULL socket and multiple PUSH sockets. With this design, each agent can act as a sink and receive messages from any other agents in the network, whilst simultaneously sending messages to other agents through the PUSH sockets. This is achieved by implementing two independent messaging queues, one for inbound messages, and one for outbound messages. This separation allows the monitoring of queue size within each agent to regulate traffic flows.

Monitoring Infrastructure
A major challenge in decentralized IoT services with autonomous social agents is to log and monitor the service, both at the individual agent level, as well as system-wide [50]. Devices and agents can be geo-spatially distributed, and deployed over different computational clusters and networks [51]. While each agent can run autonomously and independently in the network, most of the analytics require an aggregated view in real-time, whilst also allowing to drill-down and investigate the internal activities within a single agent. Providing such multi-granular views is even more challenging when considering the possible large scale of the system [52]. Simple NFS (Networked File System) solutions may not provide the necessary throughput to sustain heavy logging from several agents [52]. Additionally, IoT devices and agents can be restricted in computational resources, hence, the logging and monitoring needs to be lightweight, simple to use, efficient, and with minimal impact on real-time performance [53].
To address these challenges, this paper devises a monitoring infrastructure, comprised of the following components: A single database, containing all logging data from agents. A single logging gateway, accessible to all agents which receives the logs and commits them to the database. The main task of the logging gateway is to perform authentication, authorization, and connection pooling for the database. Finally, a single, lightweight peerlet on each agent, known as the monitoring peerlet, collects the logs and submits them to the logging gateway 13 . This infrastructure is easy to integrate (by only adding the peerlet) in Protopeer nodes. It is distributed, and modular in design, and can be connected to various observability platforms and dashboards, such as Grafana 14 and Redash 15 for real-time visualization. This infrastructure is extensively tested under high-throughput and dynamic settings, illustrated in Section 7.2

Studied Services
Based on the conceptual architecture and the realization protocol, this paper studies two live implementations of generic multi-purpose IoT services, as proof-of-concepts and use-cases of the architecture; Namely, distributed learning utilizing the I-EPOS system [5] (Section 5.1), and decentralized collective measurements using the DIAS system [6] (Section 5.2). I-EPOS performs distributed learning for multi-agent combinatorial problems [5]. The network topology of I-EPOS is structured (tree-topology) with synchronous learning iterations and communication between agents. On the other hand, DIAS performs decentralized privacy-preserving data analytics, as it relies on local computations, peer-to-peer interactions and hashed information [6,4]. It has an unstructured topology (P2P), with asynchronous learning iterations [6]. Due to their decentralized, socio-technical design, both of these services are very relevant to IoT applications. However, they are profoundly different in operation, which challenges the flexibility and applicability of the proposed architecture.

Collective Learning
For the first service, this paper studies the I-EPOS system [5] as a fully decentralized, self-organizing, privacy-preserving 13 This logging infrastructure is incorporate in the DISC toolkit. The previous version of Protopeer utilised locally saved post-process text logs, which create accessibility challenges in distributed large-scale applications 14 https://grafana.com 15 https://redash.io combinatorial optimization mechanism. The I-EPOS agents (service agents) provide the learning service, and each have a set of local plans generated by the application agent. These plans can be alternative routes from an autonomous vehicle, or power consumption schedules from a smart appliance (e.g., smart washing machine). At the system level, I-EPOS requires a system-wide goal, measured by a global cost function. This goal can be load-balancing traffic flows in a city [12], or peakshaving power demand for Smart Grids [13]. The I-EPOS agents interact and cooperate with each other to select a plan that minimizes the global cost. The agents self-organize in a tree-topology [48] as a way of structuring their interactions. I-EPOS performs consecutive learning iterations, which include two phases: the bottom-up (leaves to root) phase and top-down (root to leaves) phase. At each iteration t, agent u selects the plan p t u,s to satisfy the following optimization objective: In the above equations, |P u | is the number of plans for agent u, and p t u,s is the selected plans of agent u at iteration t. f G (p t u,s ) is the global cost of selecting p t u,s , which can be the variance of traffic load across different routes (in case of traffic load balancing). Each plan has local cost, calculated by f L (p t u,s ), which can be trip duration (in case of alternative routes), or user discomfort (in case of shifting power consumption). f U (p t u,s ) is the unfairness, calculated as the dispersion of local cost of the selected plans for all agents, with the lower values indicating more equal distribution of local cost across all agents. The α, β, and 1 − (α + β) parameters indicate agents preference for unfairness, local cost, and global cost, respectively. For instance, an agent with α = 0, β = 1 is known as a selfish agents which prioritizes minimizing its local cost, while another agent with α, β = 0 is known as an altruistic agent which minimizes the global cost. After the final iteration F is completed, p F u,s is presented to the users' devices for execution. Further elaboration on I-EPOS is out of the scope of this paper and the interested reader is referred to previous work [5].

decentralized Collective Measurement
The second studied service is DIAS (Dynamic Intelligent Aggregation Service) [6], which performs fully decentralized privacy-preserving data analytics for the Internet of Things. Application agents act as data suppliers and consumers: Data suppliers are sensors that locally generate/collect a stream of real-time privacy-sensitive data, while data consumers require collective aggregate information over the provided sensor data (e.g., summation, average, max/min, top-k.). For instance, data suppliers provide consumption data from residential smart meters, and data consumers receive the aggregated power consumption of the neighborhood, as well as status updates about the reliability of the Smart Grid [54]. This process is fully decentralized and privacy preserving, as users' data is not shared with a central entity. Each pair of data supplier and consumer is connected to a DIAS agent. Each DIAS agent (service agent) contains a disseminator and/or an aggregator. The disseminator is connected to a data supplier and discovers aggregators in the service network, to which the local sensor data is sent. This discovery is performed via a fully decentralized gossiping protocol, the peer sampling service [55] 16 . Disseminators spread the local sensor data in the network periodically by pushing them to remote aggregators to maintain a high accuracy in the estimations of the aggregation functions. Each aggregator is connected to a data consumer, and collects the input data for the computation of the aggregation functions. Finally, disseminators receive the outcome of the performed aggregation. Each bilateral peer-to-peer interaction between a disseminator and an aggregator is referred to as an aggregation session.
The raw data from data suppliers can be privacy-sensitive and rapidly varying over time. To tackle this challenge, DIAS utilizes data summarization by assigning the raw values from stream data to a selected state chosen from a limited number of k possible states. Additionally, DIAS addresses two other uncertainties: changes in the set of possible states, and agents leaving/failing/rejoining the network. The challenge here is to preserve the accuracy of DIAS estimations under these two dynamics. To address this, DIAS uses a distributed memory system based on bloom filters to track the history of the performed computations and when needed perform self-corrective actions. Further elaboration on DIAS is out of the scope of this paper and the interested reader is referred to previous work [4,6].

Experimental Methodology & Settings
The experiments in this paper are divided into two evaluation scenarios, both utilize the DISC toolkit (Section 4), and follow the conceptual architecture and communication protocol illustrated on Figures 1 and 2, respectively. The first evaluation scenario (Section 6.1) studies the accuracy of the two services in live environments, to provide a performance benchmark for the testbed architecture and DISC toolkit given experimental realism [15]. The second evaluation scenario (Section 6.2) studies the efficiency and robustness of the services during long-term operation under dynamic and volatile environments.

Evaluation Scenario I: Comparing Accuracy in Nonvolatile Environments
Experiments under live environment, even without dynamic changes, can incur some inaccuracies due to networking errors (e.g., packet losses), clock differences among machines, and system failures [7]. This evaluation scenario analyzes the accuracy of the two studied scenarios and provides a benchmark

I-EPOS
The experimental setting and parameters for I-EPOS in evaluation scenario I are illustrated in Table 2. The utilized dataset contains charging plans for 2779 electric vehicles (EV) in three different planning horizons: 1,3, and 7 days ahead [56]. In each case, every EV has 4 alternative charging plan in the form of a vector, specifying the energy demand at each minute during the planning horizon. For 1 day ahead plans, the length is 1440 (24h * 60min), and for 3 and 7 days ahead plans, the length is 4320 (3d * 24h * 60min), and 10080 (7d * 24h * 60min), respectively. Two different global cost functions are applied to the aggregate charging demand of the participating EVs, each addressing a different charging scenario: (i) minimizing charging demand variance (MIN-VAR), and (ii) shifting charging times to night (MIN-RMSE) 17 . Each plan also has a local cost, which is its discomfort calculated by the likelihood of using the EV while charging [56]. This likelihood is extracted from the historical usage data of each EV 18 . The performed experiments are based on the 12 profiles illustrated in Table 3 illustrating various configuration of the system. Each profile is tested 100 times. Overall, there are 1200 experiments in the simulation, and 1200 in the real-world environment.
To compare the performance between the simulation and live environments, the relative differences between the global cost where g t s,i and g t l,i are the global costs of profile i at iteration t in simulation and live settings, respectively. Similarly, l t s,i and l t l,i are the average local costs of all agent in profile i at iteration t in simulation and live settings, respectively.

DIAS
These experiments are based on the GDELT (Global Dataset of Events, Languages, and Tone) platform 19 . GDELT monitors the global human society and captures print/broadcast/webbased news media in real-time at a planetary scale. Its data can be accessed via an API in almost real-time, i.e. every 15 minutes. This paper employs the DIAS-GDELT demonstrator [57] 20 . It fetches GDELT news updates every 15 minutes, extracts the possible states, and sends them to the application agents. DIAS agents (service agents) are mapped to 28 application agents, each representing a country from GDELT. Each DIAS agent receives the number of news generated during the last 15 minutes from the application agent, disseminates them in the network, and receives the aggregated the total number of news generated by the other GDELT country agents.

Evaluation Scenario II: Handling System Dynamics
In this evaluation scenario, a set of continuous real-world experiments are performed to study the performance of both services under complex and dynamic environments. Each day is divided into three 8-hour time periods: low, medium, and high intensity, each imposing different rate of changing system dynamics. 19 https://www.gdeltproject.org [Last accessed: February 2020] 20 http://dias-net.org/dias-gdelt-live/ Last accessed: February 2020

I-EPOS
At the start of each day, the I-EPOS service is initialized with 200 agents. Each agent is randomly assigned to one of the 2779 EVs from the EV dataset with 7 days ahead planning horizon. During runtime, four dynamics are adopted, each corresponding to a change in system settings: (i) Agents joining/leaving, (ii) local plan change, (iii) α and β change, and (iv) change of the global cost function. The rate of change for each dynamic varies across the intensity periods, with the highintensity period incurring the highest number of changes. The rate of change for each dynamic across different intensity periods are shown in Table 4. For example, in the low-intensity period at the end of each run 21 an agent changes its plans with 5% probability. The rate of change for α and β operates the same way. However, the change in global cost function is applied system-wide. Additionally, at the end of each run in the high-intensity period every I-EPOS agent leaves or joins the network with 50% probability. The experimental settings for this scenario are shown in Table 2.
In addition, the effect of rate of change on the performance of I-EPOS is studied using two metrics. The latency indicates the variation of the I-EPOS execution time with varying dynamics, with respect to static and non-changing dynamics [58]. The execution time is defined as the time it take (in milliseconds) for I-EPOS to complete 50 iterations (an I-EPOS run), plus applying changes enforced by the dynamics (e.g., agents join/leave, changes in plans or α and β values). The latency can be calculated as follows: Latency := Execution Time with Varying Dynamics Execution Time without Dynamics The second metric is WAT, which indicates if the system is excessive time adapting to dynamic changes rather that performing service-related task. The WAT can be calculated as follows: WAT := Working time Adaptivity time (4) where the working time concerns the time (in milliseconds) required to execute the 50 learning iterations of I-EPOS (an I-EPOS run), while the adaptivity time concerns the time required to adapt to changes in dynamics [58]. For instance, changes in the number of agents which triggers the self-reorganization of the tree-topology. 21 Each run refers to the completion of 50 learning iterations by I-EPOS.

DIAS
At the start of each day, DIAS is initialized with 20 agents. During runtime, three different dynamics are adopted, each corresponding to a change in the system setting; (i) Rate of agents joining/leaving, (ii) rate of change in the set of possible states, and (iii) rate of change in the selected state. Every time an agent changes its set of possible states, it randomly selects 9 numbers between the current time and the next hour. For instance, the possible states for an agent at 14:00 (1400) is 9 number in the range of [1400, 1500]. The probability of changes for each dynamic varies across the intensity periods, with the highintensity period incurring the highest number of changes on the system. The rate of change of each dynamic is shown in Table  5. For example, in the low-intensity period, each DIAS agent changes its selected state every 5 minutes. The agent join/leave probability means that, in the high-intensity case, all agents leave the network every 2 minutes, and return 2 minutes later.

Deployment Infrastructure
The deployment infrastructure of the testbed is as follows: one server with higher computational power for scaling, and a less powerful server for long-running experiments. Both servers provide 'bare metal' access, which is substantially faster than using virtual images. The larger machine has the following specifications: Intel Xeon hex-core 3.50GHz 256GB DDR3 RAM, 2TB Raid 1 storage, Ubuntu 16.04. As for the smaller machine: Intel Core i7-6700 Quad-Core, 64 GB DDR4 RAM, 1TB storage space, Ubuntu 16.04. Each service agent is implemented using the Protopeer toolkit (Section 4) as a separate JVM object 22 . The logging gateway is a single persistence daemon, that creates a single connection to database (PostgreSQL 10.6) and has a predefined commit rate and queue size that can be adapted based on system scale. It listens to ZeroMQ messages with logging information sent by the agents, and commits the information to the database. Each agent notifies the logging gateway of the required relations, tables, and indices, in the form of SQL query templates, to be created on the database. The communication between all agents is based on message passing, implemented based on the ZeroMQ library.

Experimental Results
This section illustrates the experimental results and evaluations based on the methodology introduced in Section 6.

Evaluation Scenario I: Comparing Accuracy in Non-
volatile Environments This section illustrates the studies the accuracy of the two services in non-volatile environments based on the methodology introduces in Section 6.1. Figure 4 shows the relative difference in global and local cost between the simulation and live environments across the I-EPOS iterations. For each profile, the global cost is calculated based on the corresponding global cost function (MIN-VAR/MIN-RMSE) in Table 3, and the local cost is the mean discomfort of agents' selected plans, calculated by the likelihood of using the EV while it is charging [56]. The value of each cell in Figure 4 shows the mean across 100 repeated experiments for the given profile. For each experiment, I-EPOS   ). Note that due to lower WAT (higher adaptivity time), I-EPOS manages to complete less runs in higher intensity period, during the same time-frame.

I-EPOS
initializes random trees and assigns the agents to it. This difference in learning initialization generates small variations in the I-EPOS outcome [59]. However, the general trend across all profiles shows that as the number of learning iterations progresses to the final iteration (50), the global and average local costs of the simulation and live environments converge. After 10 learning iterations, the relative difference in global cost for all profiles is less than 0.02. The highest difference in average local cost at the final iteration is 0.0256 related to Profile 10. This confirms that transition of I-EPOS from simulation to The accuracy of this estimation is affected by the various factors, such as disseminator sampling pool size, convergence time, and the selected state changes. DIAS error is the difference between true sum (raw data) and the estimated sum. While this error increase with the rise in intensity, due to quick dissemination and convergence, the rolling mean remains low.
live, based on the DISC toolkit, can be performed with minimal introduced inaccuracies. Figure 5 outlines three time series based on the experimental methodology introduced in Section 6.1.2: (i) GDELT Actual: the raw baseline values extracted from GDELT. This represents the total number of news items generated by the 28 countries. (ii) DIAS Actual: the sum of selected states from each of the 28 DIAS agents. Each selected state is the number of the generated news items by the assigned country. The set of possible states is extracted by a sliding a window of 27 observations, uniformly sampling 9 values. (iii) DIAS Estimated: The estimated total number of news items by all countries (estimated DIAS actual), calculated by averaging the estimates of each agent. This estimation is what each DIAS agent calculates as the true value for the DIAS actual. The accuracy of this estimation is affected by various factors, such as disseminator sampling pool size, convergence time, and the selected state changes. This experiment has been running since November 1 st , 2018. As illustrated in Figure 5, DIAS service based on the DISC toolkit can perform long-running operation and accurate estimation of the GDELT baseline, and adapt to sudden changes quickly.

Evaluation Scenario II: Handling System Dynamics
This section illustrates results of the experiments in dynamic environments, based on the methodology introduced in Section 6.2.

I-EPOS
These experiments are continuously executed between November 25 th , 2019 to January 12 th , 2020, with the intensity setting changing every 8 hours. Figure 6 illustrates a snapshot of I-EPOS live operation during December 12 th , 2019. During a typical working day, the I-EPOS live handles approximately 80.000 changes in agents' plans, and α, β parameters (Figures 6a and 6b), 3000 agents joining/leaving (Figure 6c), as well as 400 changes in the global cost function (Figure 6d). Figure 6f shows the latency of the I-EPOS across different intensity periods. On average, the latency increases by 27% from low to medium, and 102% from medium to high. Figure 6g shows the WAT in different intensity settings, where the average WAT is always higher than 1. Generally, if the ratio is less that one, the system is spending a lot of time adapting to the changes [58]. The above experiments confirm that even under highly dynamic environments, I-EPOS completes its learning iterations without any crashes/failures, and provides the learning outcome.

DIAS
The experiments are continuously executed between November 23 rd , 2019 to January 8 th , 2020, with changing intensity settings every 8 hours. Figure 7 illustrates a snapshot of DIAS live operation during December 12 th , 2019. During a normal working day, the DIAS handles approximately 4000 agent joins/leaves (Figure 7a), 16,000 state changes, and 2 million exchanges of messages (Figure 7b). The estimated sum of the selected states of all DIAS agents is shown in Figure 7c. As shown, even under intense dynamic changes, the DIAS live still provides accurate estimations. Finally, Figure 7d illustrates the overall DIAS error, calculated as the difference between true sum (raw data) and the estimated sum. This error is caused by various factors, such as summarization (raw values to the set of possible states), rapid state changes, agent joining/leaving, and convergence time. As shown, this error increase with the rise in intensity, however due to quick dissemination of state changes and convergence in the network, the rolling mean error is low.

Conclusion and Future Work
This paper introduces a novel IoT testbed architecture for decentralized socio-technical services and applications running on IoT. This architecture applies two layers of abstraction on both the IoT application (devices), and the decentralized services, enabling a dual self-integration capability: (i) an IoT application integrating several application-independent and modular decentralized services, and (ii) a decentralized service integrates to several IoT applications without, changing the implementation of the service. A distributed communication protocol is designed to realize and operationalize the conceptual architecture, providing common interfaces and the communication logic required for self-integration of applications and services at runtime. Additionally, this paper contributes the DISC toolkit, providing a general purpose IoT prototyping toolkit for rapid design and testing of decentralized socio-technical applications, as well as facilitating the transition from simulation to live environments. Experimental evaluations on two decentralized IoT services, performed under highly dynamic environments with confirm the efficiency, and robustness of the testbed architecture and accompanying toolkit.
This work promises new instruments for prototyping and developing decentralized socio-technical services running on IoT, and pathways to manage their complexity, which so far have hindered value-oriented self-management approaches in Smart Cities. Ultimately, the architecture and toolkit will be able to facilitate pilot tests in Smart City use-cases. Future research can address the inclusion of other decentralized services with different requirements and network structures, device/agent mobility, and semantic service composition. Lastly, while limited access to IoT devices and computational resource narrow down the scale of the experiment to few hundred agents, applying the architecture and toolkit to existing physical testbed, or global research networks (e.g., PlanetLab 23 ) can provide further insights and improved the applicability of architecture and toolkit.

Artifacts & Reusability
To facilitate the reusability of the testbed and DISC toolkit by the community, the code bases, protocols, and the documen-23 https://www.planet-lab.org/about tations are made openly available. In addition to simulation versions of I-EPOS 24 , and DIAS 25 , the live EPOS 26 , live DIAS 27 , monitoring infrastructure 28 , redesigned Protopeer 29 , and IoT device agents 30 are also available for the community.