{'fraction_non_alphanumeric': 0.022556390977443608, 'fraction_numerical': 0.004544327852598529, 'mean_word_length': 5.017151379567487, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving lifecycle query in integrated toolchains using linked data and MQTT-based data warehousing

The development of increasingly complex IoT systems requires large engineering environments. These environments generally consist of tools from different vendors and are not necessarily integrated well with each other. In order to automate various analyses, queries across resources from multiple tools have to be executed in parallel to the engineering activities. In this paper, we identify the necessary requirements on such a query capability and evaluate different architectures according to these requirements. We propose an improved lifecycle query architecture, which builds upon the existing Tracked Resource Set (TRS) protocol, and complements it with the MQTT messaging protocol in order to allow the data in the warehouse to be kept updated in real-time. As part of the case study focusing on the development of an IoT automated warehouse, this architecture was implemented for a toolchain integrated using RESTful microservices and linked data.


Introduction
The explosive growth of IoT devices and systems is accompanied by their increasing complexity. Modern engineering environments (toolchains) required to develop such systems involve many tools and interdependent processes, such as requirements analysis, embedded and cloud architecture, prototyping, development, testing, deployment etc.
In order to facilitate these processes without impacting their quality and flexibility, some integration has to take place to allow various tools and processes to be connected. In systems engineering, one of the important reasons to perform such integration is stimulated by the need to perform a cross-system lifecycle query (LCQ). LCQ allows one to find resources that satisfy certain constraints defined across many tools in the distributed toolchain. An example query that can be relevant for a test engineer can be formulated in the following way: Find all System Components that have been designed to satisfy the Requirements with the status "APPROVED" and that have at least one failed System Test, which has not been marked as "RESOLVED".
Another reason for building such a toolchain is that with the advent of Big Data, the possibility to support and guide the systems engineering efforts for the IoT systems by the use of the operational data becomes viable. Such integration of operational data would also require advanced query capabilities across raw data and over the insights generated by applying various data science methods to the raw data.
Traditionally, a flavour of an extract-transform-load (ETL) process would be used to transform and connect tool information. Analysed against the 4Vs of the Big Data [8], one can argue that ETL limits the velocity, variety and veracity of the data. Velocity is reduced because the batch nature of ETL does not guarantee access to the most recent data. Variety takes significant effort to achieve as each integration requires an individual ETL pipeline. Finally, the transformation step of each pipeline endangers veracity if semantics are not preserved.
Instead, using microservices and Linked Data [23] to integrate systems within the toolchain eliminates the need for multiple transformations and ensures that the engineering environments of the IoT systems are integrated correctly and efficiently. Assuming this approach for the toolchain integration, one of the following architectures can be used to add an LCQ capability to the toolchain: This paper analyses the suitability of these approaches for IoT systems. Based on this analysis, we identify their shortcomings when used in the context of modern IoT systems. Instead, we propose an improved approach where a data warehousing approach is complemented with a messaging protocol to allow for a centralised LCQ capability while ensuring the data is updated in real-time. This allows the LDW to run the queries immediately after the engineering tool data has been updated.
We use a case study for the development of a robotic warehouse to highlight the specifics of such architecture. The case study involves tools for requirements analysis, change management as well as systems engineering, which have to be integrated into a single toolchain.
In Section 2, we briefly present the background information on Linked Data, MQTT, OSLC, and TRS. In Section 3, we present the case study. In Section 4, we present the requirements for the LCQ architecture and evaluate the available architectures according to them. In Section 5, we propose an improved data warehousing architecture and detail its implementation. Section 6 covers related work, followed by a discussion in Section 7 and a conclusion in section 8.

Background
Tool integration is performed between tools on different levels, such as data format, user interface, reuse of functions [32]. Using web-based Linked Data approach allows the integrations to be made in a platform-independent and scalable way [26]. Resources are uniquely identified using HTTP URIs and can link to other resources across such integration, while representing the data using Resource Description Framework (RDF) [12].
Open Services for Lifecycle Collaboration (OSLC) is a set of specifications that define requirements for the discovery capabilities of the Linked Data services, their structure and resource shapes. This allows OSLC-compliant services to be used perform application integration [10,19]. OSLC builds on top of the established web standards and best practices like HTTP, REST, RDF, Linked Data Platform (LDP) [9].
Using OSLC for building integration services allows defining domain vocabularies separately from the services. These vocabularies can be used for two main purposes: to communicate between services as well as to translate between the tool-specific schema and the OSLC service vocabulary. A common vocabulary for the services and their OSLC capabilities can be defined using modelling tools [18], while the source code implementing the REST services and OSLC resource classes can be generated from the model [17].

Case Study
Robotic warehouses are used to store and forward goods and require a range of equipment (such as conveyor belts, identification and tagging systems, selfdriving vehicles and robots) in order to perform various logistic tasks (Fig. 1). Logistics automation is an activity for optimising warehouse processes to improve its efficiency. The efficiency of logistics in the robotic warehouse heavily depends on the "intelligence" of a deployed IoT system. Therefore, the IoT system's software needs to be continuously improved to optimise the warehouse operations. Such improvements need to be supported by an efficient development toolchain. The toolchain in the use-case integrates the following tools: 1. A Requirements Analysis tool to capture functional and non-functional requirements. In the use case, Eclipse ProR [11] will be used for this purpose. 2. A Design tool to develop various parts of the IoT system. In the use-case, Matlab Simulink [13] will be used to support model-based development approach. 3. A Change Management tool to ensure that when bugs arise or requirements change, the IoT system is changed a controlled manner. In the use-case, Bugzilla [4] will be used for this purpose.
Originally, these tools have no built-in integration between them, even if they expose a certain API. However, entities in each of these tools are naturally related to the entities in the other tools. With no integration between them, such relationships are implicitly defined. For example, a change request CR1 in the Bugzilla tool may refer a requirement R1 in the text description, which does not allow to define an explicit traceability link between the resources. As a precondition to performing an accurate LCQ across the toolchain, an integration capability is needed to make those links explicit. Overview of a Linked Data Toolchain, clarifying the distinction between its Integration (above) and Query (below) capabilities.
The Linked Data Toolchain presented in the use-case consists of three logical parts ( Fig. 2): 1. A set of tools that make up a toolchain.

A set of corresponding
Linked Data services, which provide an integration capability for the toolchain. 3. A Lifecycle Query system, which provides an LCQ capability.
Linked Data services expose all entities managed by the underlying tools as RDF resources via an OSLCcompliant RESTful service. The rest of the paper assumes the that the integration capability exists and focuses on the LCQ capability.
To evaluate different LCQ architectures, we define a number of analyses that should be incorporated into the development process.
LCQ1 First, we want to ensure that every design artefact was developed because a requirement demands it: List all Simulink blocks that are not linked to any Requirement.
LCQ2 Second, when considering a change request on a requirement, a requirements analyst needs to assess the impact of a requested change on other pending changes to the related requirements: For a given Change Request CR1 linked to the Requirement R1, list all Change Requests CRx linked to the Requirements Rx that refine the requirement R1 as well as all Change Requests CRy linked to the Requirements Ry that are refined by the requirement R1.
LCQ3 Finally, when a requirement or a design artefact does not change at all in a big project, it is also suspicious and should be checked: List all Models and Requirements which are not the subject of any Change Request.

Analysis of the lifecycle query architectures
In this section, we analyse the LCQ capability needs for our case study, to deduce and formalise the requirements they places on any LCQ architecture (subsection 4.1). Afterwards, in the subsection 4.2 we evaluate how different LCQ architectures satisfy the defined requirements.

Lifecycle query requirements
In the distributed Linked Data approach used for tool integration, the lifecycle query capability has to satisfy a few constraints.

RQ1
In order to ensure that the LCQ results are not obsolete by the time the queries are executed, LCQs should not run on outdated data that differs significantly from the data operated by the tools. The lifecycle query system shall minimise the time difference between the age of data used by the integration capability and the LCQ capability.
RQ2 One aspect that impacts the end-user usability of an LCQ capability is the query performance itself. An LCQ that can deliver real-time performance can be better integrated with the tool interfaces, making it more valuable for engineers. A non-realtime but relatively fast LCQ can be used for notifications. Finally, very slow queries can only be delivered as reports.
The lifecycle queries should execute in minimum amount of time.
RQ3 Sometimes, the execution of many queries can introduce a lot of load on the system. LCQ capability should not overload the integration capability.
The load the LCQ capability puts on the rest of the toolchain shall be minimised.
RQ4 Large development processes span a wide range of tools, which can lead to a big toolchain. The LCQ capability shall be able to scale accordingly.
The LCQ capability should efficiently scale to support larger toolchains.

RQ5
A toolchain configuration will change over time, where tools may be added, removed or updated. For this reason, the development of a new tool service to provide services to the LCQ capability shall not be prohibitively complex.
Total effort needed to enable the LCQ capability for an individual service should be minimised.
RQ6 Changes in business requirements, development processes, and engineering tools may introduce a need for new or modified queries to collect new metrics or verify a certain state of resources. The LCQ capability should be flexible enough to easily adapt to the new requirements.
Effort required to add a new LCQ or modify an existing one shall be minimised.
RQ7 Toolchains include legacy tools and implementing correct integrations requires a lot of effort. An integration capability based on Linked Data also provides an additional benefit of being tool-and implementation-independent. The LCQ capability should fit into the toolchain well and should not nullify those efforts.
The LCQ capability should require minimal architectural changes to the toolchain and its integration based on Linked Data.

Lifecycle query architecture comparison
Given the toolchain architecture, with an integration capability based on Linked Data, the following ways can be used to implement the LCQ capability: 1. Direct Query over REST, where the data is gathered manually through a series of HTTP requests to the respective services of each tool that holds the data to be queried. 2. Distributed Query using the SPARQL Federated Query. 3. Data Warehousing solution with a centralised query capability.
While the Direct Query gives the developer most control over querying and ensures that the data is always up-to-date, the developer must determine the request order on a case by case basis (query planning) and make sure that the data is cached properly, in order to avoid overloading the integration capability. Even with all these concerns handled, developing new queries is non-trivial, whether the developer decides to develop the queries programmatically or introduce a simplified domain-specific language (DSL).
Switching to the Distributed Query, one can rely on the query planner of a triplestore. The approach, however, has its own limitations: -The triplestore of a specific service may cache responses from the certain calls but cannot determine the caching validity of the underlying RDF resources (as opposed to the manual query that can and should rely on the Cache-Control HTTP header for its REST calls). In particular, the SPARQL Federated Query specification does not cover caching and only allows to silently ignore errors to prevent cascading failure of a query [14]. -Similar to Direct Query, Distributed Query causes an increased load on the toolchain. The load has now instead shifted from the microservices to their triplestores. The expressiveness of SPARQL can allow an even higher load on the toolchain, which can lead to the SPARQL endpoint downtime [6]. -Exposing a database is an anti-pattern, creating problems for access control [15] as well as limiting the use of patterns that rely on reified statements and named graphs [16]. -Writing Federated Queries is more cumbersome than plain SPARQL and expects the developer to statically define the query endpoints (using the SERVICE keyword). In most cases, the UI would have to be developed to abstract the query formulation from SPARQL.
Finally, the Data Warehousing appro-ach allows queries to be developed using plain SPARQL without a concern for caching, error handling, or the distributed nature of the tool-chain (including the risk of overloading the tool-chain services with "heavy" queries). The main challenge is to keep the data in the LDW updated with every service in the toolchain.  Table 1 summarises how different approaches satisfy the original requirements of the lifecycle query. It uses the following comparative values: -"++" denotes that the requirement is fully satisfied -"+" denotes that the requirement is partly satisfied -"-" denotes that the requirement is poorly satisfied, additional work may be necessary to overcome the shortcoming Each architecture has its own strengths and weaknesses, but in general, Data Warehousing architecture has the best fit for the LCQ system. Data Warehousing architecture has, however, two important issues: 1. In most warehousing solutions, data in the LDW is updated through regular pull requests for changes to each of the tool services. Depending on the frequency of these updates, a difference in the data state will exist between the LDW data and the original data in the toolchain.

Integrating a new tool in the tool chain requires additional implementation
in the corresponding Linked Data service to support LDW requests.
While acknowledging the suitability of the data warehousing approach, we present in the next section an improved architecture that addresses these identified issues.  The TRS client is the core component of the LDW solution. It implements the TRS protocol, which allows a server to expose resources so that these changes can be discovered and tracked by the clients over HTTP. Its main responsibilities are:

ProR
initiating the synchronisation, fetching all subsequent resource update events, compacting the update event list in order to avoid applying multiple changes on the same resource (ie if the resource was created, modified, and deleted, nothing will be done at all). Now, the TRS protocol implies a pull-based periodic approach to fetch updates from the servers. The update period is defined by the TRS client and can be adjusted per TRS server, depending on the frequency of changes to the underlying resources. This, nevertheless, leads to a delay between the source data and the information in the TRS client. To remedy the staleness issue identified in the previous section, we complement the TRS protocol with MQTT to eliminate the polling period and shorten the LDW data update delay.
MQTT is a protocol to allow publish-subscribe-based messaging [31]. A message broker is used to receive, store, and distribute messages between clients. The small footprint of the protocol made it attractive for the use in IoT systems, particularly gateways [2,3,7].
Upon a change in a tool, the TRS server sends a ChangeEvent message via MQTT containing 2 properties of a single change event (event sequence number and the URI of the changed resource) and its type (one of Creation, Modification, Deletion, as defined in the TRS specification). The Resource Updater then updates the RDF resources in the triplestore after reconstructing the TRS Change Event from the ChangeEvent message.
In this architecture, any LCQ queries can be performed across the entire dataset of the toolchain using SPARQL queries on a single endpoint.

Implementation
The LCQ architecture presented in this paper was implemented and integrated with the toolchain used in the case study. The implementation consists of the lifecycle queries, which are available on Github * , and 2 open-source modules, which have been contributed to the Eclipse Lyo project † : -The LDW service has been contributed as a TRS Consumer.
-The implementation of a TRS Server has been extracted into a library that supports the REST API needed communication with the LDW in a JAX-RS service.

Analysis
In order to assess how the proposed architecture compares to the existing ones, we will reevaluate how it satisfies the original requirements. RQ1 The proposed architecture has still greater latency than the Direct Query (DRQ) and Distributed Query (DSQ) approaches but improves significantly against the Data Warehousing (DWH) architecture without messaging. An important note here is that the DRQ and DSQ approaches cannot achieve zero overhead because they communicate with the linked-data-based OSLC service instead of directly communicating with the tool. Depending on the tool integration method, the difference between the proposed architecture and the DRQ/DSQ approaches might be minimal.
RQ2 The proposed architecture makes no improvements compared to the base DWH approach but takes advantage of its centralised query capability.
RQ3 The proposed architecture fetches each resource state at most once (some retrievals can be avoided due to change log compaction) and is therefore optimal in this regard. However, the big volume of small updates can increase the system load. For such cases, both the TRS Client and the TRS Server library can batch a certain number updates from a service.
RQ4 The publish-subscribe paradigm used in the proposed architecture is inherently more suitable for building scalable distributed systems [20].
RQ5 The TRS Server library drastically reduces the number of source lines of code (SLOC) needed for the implementation of a compatible interface. It requires more effort than DRQ but less that DSQ, especially if the OSLC service is not using a triplestore as an underlying data source, but a database like MongoDB or simply caches the Linked Data converted from a tool in a key-value storage like Redis.
RQ6 The DWH approach provides an ability to write simple SPARQL queries and our proposed architecture does not affect that.
RQ7 The proposed approach does not affect any of the existing Linked Data architecture but introduces a messaging system, which might exist in many enterprise integration platforms already [24]. The problem arises when the source of the OSLC service cannot be modified (especially when a proprietary tool comes with a ready OSLC implementation, such as IBM DOORS NG [1]). To that extent, our architecture allows those tools to be integrated into the LDW using an unmodified TRS protocol.

Future work
First, we would like to evaluate the proposed architecture on a toolchain with a production load and identify ways to improve its efficiency further. Potential solutions to improve performance include: sending updated resource contents in a message together with a change event, publishing the base set of resources in a compressed binary HDT format [21]. Additionally, an OSLC service can modified to stream change events immediately when the resource updates are received from the underlying tool.
Finally, we would like to explore an architecture where each OSLC service has both a TRS server and a TRS client components. This would allow the resource changes to be distributed across all services in the toolchain, not only to the TRS client of an LCQ system, which would make the toolchain reactive.

Related work
Efforts to enable query over distributed data sources predate Linked Data. Sheth and Larson have provided a taxonomy of multi-database systems and relevant concepts [30]. Levy et al. [25] proposed the generation of query plans from complex queries that detail the individual queries to be executed against the data sources and the rules for combining the results of such distributed query. Support for distributed queries over RDF resources has been implemented in the SPARQL 1.1 Federated Query [22], but the query developer is responsible for specifying the subqueries against other triplestores using the SERVICE keyword. Quilitz et al. introduce DARQ, an engine that allows hiding these subqueries by equipping the triplestore with the service description that is used for query rewriting [29]. DARQ has serious limitations and has been discontinued [5].
SDShare is a protocol allows RDF resources to be synchronised between servers [27]. The changes are published using Atom feeds [28]. A collection of resources is described by a feed containing a snapshot of resources at a given time and a feed containing the updated fragments of the snapshot. Compared to TRS, it lacks a concept of a cutoff event that allows rebuilding the TRS Base as opposed to the fixed SHShare Snapshot, among other things.
In an effort to make SPARQL scalable and highly-available, Linked Data fragments (LDF) have very promising results and represent a viable approach to "cloud-scale" SPARQL [34,33].

Conclusion
In this paper, we presented the need for the LCQ capability in integrated engineering environments. We also presented a case study involving a toolchain integrated using OSLC-compliant microservices and the requirements that such toolchain puts on the LCQ capability. Three LCQ architectures were presented and evaluated according to the identified requirements.
An improved LCQ architecture was proposed, which complements the TRS protocol by delivering change events in messages over the MQTT protocol. This allows reducing the delay between the original resource update and the update of a corresponding resource in the LDW. The proposed architecture has been evaluated according to the requirements and compared to the previously discussed ones. The architecture provides advantages over the plain LDW as well as over the Direct Query and the Distributed Query architecture.