{'fraction_non_alphanumeric': 0.020940254228112347, 'fraction_numerical': 0.005348751597046277, 'mean_word_length': 4.999740158503313, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DevOps and Microservices in Scientific System development

There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware platforms, combined with the increasing multidisciplinarity of science itself have exacerbated the problem because self-taught"end user developers"are not familiar with the disciplines needed to tackle this increased complexity. On a more positive note, agile programming methods have approached software development practices to the way scientific software is produced. In this work, we present the experience of a multi-year industry research project where agile methods, microservices and DevOps were applied. Our goal is to validate the hypothesis that the use of microservices would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration. Hence, scientific multidisciplinary systems would gain in a twofold way: (i) Subject Matter Experts(SME) use their preferable tools to develop the specific scientific part of the system; (ii) software engineers provide the high quality software code for the system delivery.


Introduction
"Simple things should be simple, complex things should be possible." This maxim by Alan Kay is considered a guiding principle by designers of programming languages, libraries, applications, and any other artefacts that might be considered to be tools. But it begs the questions: Simple or possible for whom? What is simplicity or complexity? Specifically in the domain of information systems, who are the tool designers and who are the users?
In this work, we consider ourselves, the software engineers, as tool designers and computational scientists as "professional end user developers" [1]. Contrary to our intention as software engineers, extensive documentation in literature 1. Increase in incidental complexity: As science itself is evolving and requiring more interdisciplinarity, so does scientific software. It is now common to couple physical models developed by different research disciplines. An example is the simulation of fluid flow in a petroleum reservoir coupled with the models of chemical reactions. While scientist in the past might have been able to "code" on their own, now they must integrate with the work of others. 2. Increase in accidental complexity: It is the nature of scientific computing that any advance in hardware is taken advantage of to run more accurate models. During the years of Moore's law, scientists could rely on increasing CPU speeds for more computing power. With the end of this trend the adoption of heterogeneous architectures with multicore processors, distributed computing, and specialised co-processors like GPUs have become a necessity. Scientists however are ill-prepared to deal with this new reality.
In the past, upfront design software engineering practices were at contrast with scientific development practices. A theory about specific phenomena is developed by a process of formulating hypotheses and running experiments to try to invalidate those hypotheses. Whenever a contradiction is found, the model must be adjusted and subjected to further experiments, resulting in an iterative process. Since scientists view their code as a representation of their evolving theory, it is clear that no complete specification of the software can be written upfront.
However, in recent years, software engineering has adopted more iterative approaches like Agile, to mitigate the risk that a software is found to be at odds with real-world requirements after years in development. Several studies applying agile methodologies to scientific computing have been documented [9,10]. Practices such as continuous integration (CI) foster an experimental approach to development where small incremental changes are tested under real conditions with staging, A/B testing, and extensive monitoring [11]. Moreover, the dynamism of development and stability required for operation generates conflicts among areas, teams and goals, demanding changes in people mentality and reasoning about the impacts in organizational work culture [12]. Therefore, tools and methods are required to meet the needs of scientists and engineers in scientific system development.
In this work, we explore the development of scientific system applying recent software engineering trends such as microservices [13] supported by DevOps [14] practices and as-a-service delivery. We support our findings on our experience of applying these trends from the start to a large applied research project involving a multidisciplinary team of more than 30 people over 3 years. In this project, in the domain of petroleum geology and geophysics, several microservices were developed by small teams composed of subject matter experts (SMEs) and developers. DevOps, CI and user interface were handled by our software engineering staff.
It is a known property of microservices that they promote a different trade-off in the complexity of the overall system: the individual components become simpler, and the integration of parts becomes more complex [15,16]. The hypothesis that we wanted to test in this project was that this trade-off would allow us to relieve SMEs to some degree from the burden of dealing with the complexity of software and shift it to the software engineering staff. The SME would use the programming language that is most familiar to scientists in her domain of expertise and her work would be integrated into the larger application using standard interfaces. In other words, we expected that microservices would allow us to re-distribute the complexity so as to minimise the overall difficulty. This paper is organised as follows. Section 3 presents how we have implemented microservices, DevOps, and CI. Section 4 presents the quantitative and qualitative metrics that we have gathered about the prototype and the teams that implemented it. Section 5 discusses some related works. Finally, section 6 presents our conclusions.

Incentive structures in scientific computing
According to Storer [3] and Johanson [2], the value of software for scientists lies solely in the amount of publishable scientific results that it generates per calendar time. Therefore shortcuts are taken to reduce the time to scientific results, be it by optimisation that reduce running time but also reduce portability or reuse, or by not investing time to modularise the code. The grant-based funding in academia c'ontributes to this because it casts uncertainty about the continuation of lines of research in the future. It also makes it hard to justify the employment of software engineers to help the scientists although it has repeatedly been demonstrated that it improves outcomes [17].
As in other industry research institutes, we do have a permanent software engineering staff that supports and orients the development of projects. We also have a long term research strategy and, therefore, value the re-usability of the software artifacts that are produced.
However, as is the case in academia, the performance of our scientists and the ones in the sponsoring organisations is still measured by metrics like the number of patents and papers produced. This means that new features tend to be valued more than quality and performance improvements of existing code. And as in academia, researchers are already very busy keeping up with their fields and tend to resist learning more advanced software engineering concepts.
As a consequence, the engineering team has had difficulties in controlling the proliferation of ad hoc solutions. As a concrete example, to reduce the size of arrays stored in a MongoDB [18] database, the native representation of arrays was exchanged by a binary blob of serialised (pickled) Python arrays. Although it reduced storage requirements in half, it made the database effectively unreadable for other programming languages, including other versions of Python. This is a good example of something that looks like an easy solution, but hides a potentially large "technical debt". We have had other instances of this problem such as in the serialisation of a machine learning model for image classification.
As software engineers, we should improve efforts to explain to stakeholders how the fast addition of features correlates negatively with code quality and therefore stability, performance, and reliability of software artefacts can suffer. Because without this awareness, end user developers might miss the chance to seek advice before committing to a bad design.

The project development
This section presents how the project was developed.

The Project
The project that is the object of this study was a multi-year joint research agreement between our research institute and an Oil&Gas company. On the client side, we collaborated with a team of geologists and geophysicists. On our side, we had experts in fields such as optimisation, computer vision, machine learning and, visualisation. Our experts had the support of interns and our software engineering staff.
The project aimed at integrating research in several different aspects of petroleum geophysics. Each of these aspects, that we called workstreams, had an assigned team which worked quite independently, therefore warranting the use of microservices.
The client SMEs did not directly participate in software development, but oriented the research and evaluated results.
The system was delivered to them as-a-service prototype with a web-based user interface. We effectively employed CI to allow quick iterations.

Architecture
A microservice architecture comprises independently developed services that work together to achieve a specific purpose [15].
Microservice architecture has become the established approach for a huge set of modern applications, as for the well-known companies like Netflix and Amazon. It is also an approach that should be considered for a new software lifecycle of a legacy enterprise application or its redevelopment. Microservices can meet the new demands and responds as flexible as possible to changes in the future. It is able to handle this in the technical and the process level serving as the basis for future concepts [19].
The general architectural pattern for our microservices is to expose a RESTful [20] HTTP interface. The advantages of using HTTP as a communication protocol are: 1. Language Independence: all popular programming languages have libraries for HTTP clients and servers; 2. Human readable: HTTP requests and responses in text format allow developer inspection without specialised tools; 3. Tooling: there is a large amount of tooling available for HTTP, from web browsers to server software, load balancers and proxies.
There also are several downsides: 1. Text-based: when transmitting data that is not natively represented as text, client and server spend CPU cycles converting data to text. Caching can help to mitigate this overhead; 2. Latency: every new request has to go through the three-way HTTP handshake and, slow start. With HTTPS it gets even worse. This can be mitigated using Keep-Alive, but only when the cost can be amortised over several requests; 3. Bandwidth: text is much more wasteful in bandwidth than compact binary protocols, if not using compression; 4. Interaction pattern: the only interaction pattern is request-response, which limits more complex interactions.
All microservices are written in Python, Javascript or Java. Horizontal concerns like request authorisation, logging, encryption (Transport Layer Encryption 2 ), load-balancing and caching are all implemented by putting a nginx server as reverse proxy in front of these microservices. This approach, besides the advantage of uniform request logging and centralised security, is one of the possible complexity trade-offs that allow us to simplify the implementation of microservices and transfer the burden to the integration team.
Since we do not operate at a large scale and all microservices are part of the same application, all microservices share the same replicated nginx servers, but we could equally well use the sidecar pattern [21] and deploy one nginx for each microservice. This would lead to a greater configuration complexity but would give us fine-grained control over individual services.
Authentication is handled by one API endpoint of a specific microservice that returns JWT tokens [22]. A script written in Lua and run by nginx then enforces that all other requests include this token in a HTTP header.
In Python, on the developers machine we use the Flask HTTP server library, but in production we switch to the uwsgi server software that provides concurrency by starting multiple Python processes and also supports the WSGI binary protocol to communicate more efficiently with the nginx reverse proxy. This is yet another example of an effective complexity trade-off.

API Gateways
In several instances, the API exposed by individual microservices is too fine-grained for the UI to consume. Therefore we have some services that act as API gateways [15], as shown in Figure 1. These compose operations into larger ones that are called by the UI. This also shields the UI from API changes in the "internal" APIs, which is good because the team that develops it is not typically involved in the back-end services, while the other teams share the responsibility for the gateway service.

Long running tasks
Some microservices in the project must execute numerical algorithms on large seismic data. In production we have seen operations that took up to 6 days. Clearly, the HTTP protocol was never meant to handle responses with such a large delay. The pattern we use to solve this is to return immediately a token that identifies the operation and start the operation in a background process. The token can then be used to query the state of the task and to obtain results when ready.
To implement this pattern, we use an encapsulated task queue in each of the microservices that need it, as shown in Figure 1. More specifically, we use the celery library for Python that handles the submission of tasks by master Figure 1: Microservices in detail processes to worker processes over a queue server like RabbitMQ. Although each task queue is logically encapsulated, for deployment we use a single RabbitMQ server to host all task queues.
The combination of RESTful microservices with asynchronous background tasks has led to significant challenges to maintain encapsulation when the execution of a task must be composed with calls to other microservices. The typical pattern that causes this difficulty is when a new request comes to the API Gateway, which in turn starts a long running task in service A and has to wait for it to end to compose the request with a book keeping operation in service B. The ad hoc solution we have thus far employed is to start so-called monitoring tasks that wait for the end of background tasks and then finish whatever needs to be done.
However, this approach has proved difficult to maintain because of fault-tolerance issues. A simple solution would be for A to call B directly at the end of the task, but this would break encapsulation and add circular dependencies between services, but it could be made to work adding the proverbial level of indirection through call-backs. We are still studying alternatives.

Data processing pipeline
In the last year of the project a data processing pipeline was added to the project. We mention it because it follows a different architectural pattern than the other services. Pieces of data, submitted individually or in batches are submitted to a queue where they are picked up by services which in turn extract and transform pieces of data and submit their results to other queues and so on. Some services just take the data off a queue and store it in a database, thus ending the pipeline.
In this pipeline, the only HTTP service is the submission service. From then on all communication between parts is done over Kafka topics, which are a hybrid of queues and publish-subscribe. The advantage of Kafka in this case is that workers can be scaled out and that we can re-play events in any depth of the pipeline due to the persistence of messages.

Infrastructure
While in the past the primary mode of delivery of our research group had been to install our software artefacts directly on the user's workstation, this is the first large project that we have provided entirely as a service. As a consequence we had to allocate and maintain an infrastructure and develop a deployment strategy. While the deployment process is the subject of Section 3.4, this section discusses the infrastructure tools used.

docker
The project started in 2015, a time when container technologies started to gain more adoption in the industry [23]. So when the first microservices were assembled from the proof-of-concept scripts that had been developed we started to package them in containers.
The adoption of container technology, and docker [24] in particular has brought us many benefits. It provides a standard medium to deploy software. All the Dockerfiles have been written by our software engineering team and we have written them in ways that allows the software to be configured by changing environment variables or overriding configuration files at run time. But we had to work closely with the microservices teams to remove any hard-coded locations of files and services.
However, there have been some difficulties as well. Some characteristics of the container model have been hard to grasp for researchers without a computing background. In particular we could cite the ephemeral nature of containers. In several cases we found that developers were writing to hard-coded locations in the container file system instead of notifying the SE team that extra volumes would have to be allocated and attached to running containers. In general the whole lifecycle of containers, from the docker build, to the registry, the docker graph, and, finally, the instantiation as a container contains several pitfalls for non-experts.
Operationally, the lack of maturity of docker has caused us problems on several occasions. It happened that the on-disk representation of the docker graph had changed after an upgrade and we had to erase everything to start again. Another problem were deadlocks in the docker daemon that forced us to restart everything.
According to literature surveys [3], small changes in libraries or even compilers used to build scientific code are often responsible for changes in the results, and therefore increase the difficulty to independently reproduce scientific results that rely on software. Virtual machine images built automatically in accordance with the infrastructure as code [25] methodology could be a solution for this problem. Studies also found that docker images perform this role equally well, if not better due to the smaller size and overhead compared to VM [26]. In addition, since there is no virtualisation involved, docker containers have direct access to hardware such as GPUs or Infiniband when running on bare metal server.

docker-compose
For several months we used a tool called docker-compose to orchestrate our containers. It was very convenient to start or stop the entire prototype with a single command. And since we were not effectively using container tags at the time, we thought it was convenient that the tool checked automatically if there was a new version of the image in the registry.
Another useful feature is that it can allocate a private virtual network on the host machine so that all containers run inside a single segregated addressing space. This was especially convenient to us when we started to run several environments at the same time, as explained in Section 3.4.
However, as the prototype grew too large for a single server we had to start looking for alternatives.

kubernetes
Kubernetes [27,28] was the container orchestration we chose to scale the project from a single workstation to a small server farm. Docker Swarm at the time was too restrictive in features and Mesos Marathon was perceived as more complex on the basis of being more than a container orchestrator.
Despite the steep learning curve at the beginning, our experience with Kubernetes has been a positive one. Multicontainer applications are created by submitting declarative specifications to a centralised etcd database. Several different kinds of controllers continuously monitor events to maintain an overall cluster homeostasis by reacting to different kinds of events. One kind of event is the submission of new specifications, other is changes in infrastructure like the introduction or removal of computing nodes or the failure of running containers. This approach to orchestration results in a system that is always trying to converge to what was specified by the user, resulting in good fault-tolerance (to the point of hiding errors) and a clear path to implement continuous integration.
Other essential features are the ability to create namespaces (that we use to deploy several versions concurrently), name resolution based on DNS, and a flexible approach to storage provisioning.
The main drawback of kubernetes is that it draws heavily on concepts and terminology of systems engineering. Its low abstraction level can potentially be an obstacle for end user developers. However, since kubernetes is offered by many cloud [29] providers, and also supports batch workloads, it could be a good alternative for researchers who have limited access to high performance computing resources. Instead of creating clusters based on virtual machines, it could be more cost effective to run multiple containers [30].

DevOps and CI
DevOps (a portmanteau of "development" and "operations") is a software development method that extends the agile philosophy to rapidly produce software products and services and to improve operations performance and quality assurance [14]. DevOps tries to improve the integrated planning and execution of delivering new value with several guidelines: (i) Make developers production-aware and vice-versa; (ii) Frame developers and operators adopt the same toolset; (iii) Short the release cycles so that the impact of releases is more manageable; and, (iv) Promote tools to automate the setup of environments to make them reproducible. DevOps aims at a continuous pipeline of delivery where new features are automatically tested in the correct environment and then approved for production.
In past projects we were already using some DevOps approaches like using standardized build and testing environments supported by virtual machines to reproduce the client's environment with as much fidelity as possible.
However, this project was the first where we fully embraced the Infrastructure as Code (IaC) [25]. Our strategy is heavily based on the git distributed version control tool, both on the microservice and the integration level.
After a few iterations we settled on having three different versions of the prototype running at the same time. One called the dev environment would run the latest version the microservices with no guarantees of API stability. The second environment, stable would be used for integration testing and the teams were expected to maintain backward compatibility in their APIs or coordinate changes with their clients. Finally, the prod environment would be exposed to our research partners (the client).
This strategy was supported at the microservice level by an agreement of all developers to maintain at least three standard branches in their repositories: master, stable and prod. All these branches would also contain Dockerfiles to build images from them. At the same time, we configured a Jenkins CI server to watch these branches for changes. Every time a change was detected, a Jenkins job would check out the latest version, build the container image and submit it to a central registry. The naming scheme adopted was registryserver/microservice_name:git_revision to create a mapping between images and git revisions.
At the integration level we created a kubernetes configurations repository. To automate the task of submitting the configurations to kubernetes, we developed a simple Python script around the kubectl CLI that would not only send the declarative YAML configuration files to the kube-apiserver but also create configmaps from configuration files and ensure that the services would be restarted if the configuration changed. It also supports environment-specific configurations by leveraging the Jinja2 templating library to bind configuration variables, but in general we tried to avoid differences between versions as much as possible. This orchestration repository had six branches: dev, stable, prod, dev-ci, stable-ci, prod-ci, as shown in Figure  2. Each pair of env and env-ci branches worked in tandem to support CI to the three environments. The main branch was meant for human operators to edit, while the *-ci branch was maintained by a Jenkins job. In practice, every time a developer committed code, it would trigger a container build job which in turn would trigger the deployment job that would do the follow: (i) Take the latest revision of the *-ci branch; (ii) Merge the latest changes from the main branch; (iii) Edit the line containing the image name of the microservice to reflect the latest revision; (iv) Commit and push the changes; and, (v) run the deployment script. Another job consisting of steps (ii), (iv) and (v) could also be triggered by changes to the orchestration repository.
We found that the advantage of this scheme is that we can track all changes to the global state of the prototype because the orchestration repository tracks the revisions of the microservices. This scheme also put the decision of publishing changes to any environment entirely in the hands of the microservice developers. Nevertheless, this scheme is probably only applicable when all teams are part of the same organization, as is our case.
Despite the successful application of CI to program code, there is one significant challenge that we still have not addressed: the CI of data and data schemata. One of the microservices uses RDF [31] to represent domain ontologies and data, which are stored in a triple store. Although the ontologies are modeled using OWL [32] and despite the fact that these files are also stored in a git repository following the same branch scheme, we have not been able yet to develop a way to apply ontology changes automatically. In other databases in the project the problem also exists, but the data schema is much simpler and has been very stable over time.

Logging, Monitoring & Debugging
It is generally accepted that enforcing correctness of software is more difficult in distributed systems than in singleprocess applications. We argue that in microservices it can be even harder because every team is free to use the languages and tools that they deem best for the task. In this scenario, the usefulness of specialised debugging tools is restricted to the cases where a developer can reproduce the error in a controlled environment on her development machine, and her tools and debugging knowledge is not necessarily applicable to other microservices.
There are seven accepted techniques that can help in reducing errors in distributed systems [33]: testing, model checking, theorem proving, record and replay, tracing, log analysis, and visualisation. Apart from automated testing that some of our microservices implement in their CI pipelines, we employ record and replay, some tracing, log analysis, and visualisation.
The foundation of our strategy is to continuously record application logs and performance metrics on all servers. We accumulate these logs and metrics in a database, elasticsearch, and make them available for searching in the form of graphs in a web-based user-interface. We employ a logging strategy for containers [34] which consists of creating a log entry for each line written to the standard output. The downside of this strategy is that multi-line execution traces are dismembered, making it harder to read them.
We have implemented custom log parsers using logstash log aggregation tool to enrich the logs and allow advanced filtering based on the environment, the microservice, and the server. In some cases we also add additional parsers to components like nginx, to extract rich information, including timing data about every HTTP request. This data supports not only debugging but also the identification of performance problems and rich visualisations. Since our services all use HTTP, we can use these logs to replay calls. For requests coming from the user, nginx also generates a unique trace identifier that is passed in subsequent requests, allowing us to group all background service calls resulting from a single user interaction.
One difficulty has been to convince novice programmers and SMEs to be consistent in their use of logging. In a as-a-Service deployment good logs are especially important for post-mortem analysis and replays, since the programmer has no direct access to the running service. Experienced programmers who have been exposed to libraries, such as log4j, recognise the benefits of such an approach and produce more useful logs. Ideally we would like to adopt a single logging discipline across all microservices, although from the point of view of SMEs this count as unwanted accidental complexity. Our qualitative survey in section 4.1 support this view since less than half of the developers of the project used structured logging approaches, although more than half declared that logs were essential for debugging.
Testing in scientific computing is challenging because it is difficult to develop a test oracle independent of the software under development [3]. Storer [3] reports that weather scientists test their code on models of perfectly spherical planets for which an analytical solution is known. We have seen this exact testing methodology on other projects. However, in this project, a great challenge for some microservices was the dependence on large datasets for testing that cause very long testing times. When integrated into an automated CI pipeline, this caused the running time to be so long as to defeat the purpose of continuous deployment. If a developer needs to fix a small bug, say a typing error, and the tests could take over an hour to run, the natural tendency is to skip testing altogether. However, we have seen several developers creating unit tests for self-contained portions of code.
A testing difficulty is the reliance on other services. Mocking the other services is a strategy that was not adopted in this project because, during its lifetime, there have been several cases of complete API re-designs, which made it hard to justify the investment building and maintaining mocking objects. Another strategy that we could have adopted is to setup entire environments with databases initialised with reference data, but we could not find the time to do it with the fast-paced development of the project. What some microservice developers ended up doing was running automated tests with their service pointing to the services of the stable environment (described in Section 3.4).
For one of our microservices that has faster running tests the team has implemented an automated API verification based on parsing the OpenAPI specification files for that microservice.

Mixed-method study
The goal of this study is to investigate the effects of the DevOps practices and tools on the development of a large scientific information system. We conducted a mixed-method study composed by a questionnaire and static source code repository analysis along with some usage metrics. The static analysis of source code repositories aims to provide an overview of the project implementation as well as another source of evidence using data in a triangulating fashion, because the sample size is small.

Qualitative survey
According to the design types described by Kitchenham and Pfleeger [35], the self-administered questionnaire was a case control study, since we asked "participants about their previous circumstances to help explain a current phenomenon". In this particular case, we asked them about their background, their knowledge on software engineering practices and tools, focusing on DevOps and microservices. We elaborated the questionnaire (Appendix A) as a web form and sent an email to the members of the project asking them to answer the questionnaire. The response rate was 18 out of 29 members of the mailing list (approximately 62%), although some people on this list had very limited involvement in the project. The questionnaire was not sent to the SMEs of the client company.
Out of 18 responses, 15 people declared to have a computer science background. The other 3 had formal training in design. In the group of computer scientists, only 2 declared to specialise in software engineering although a total of 12 rated their software engineering expertise as high or above average. The other specialised in a diverse set of areas like human-computer interaction, databases, computer graphics, multimedia and computer vision. Only 6 persons reported to have worked on code bases of more than 50K LoCs. 8 had previous experience with agile methodologies.
When asked to rate their involvement with development on a scale of 1 to 5, 15 people chose 3 or above. These people, who we will call developers, were not exactly the same 15 people with computer science background. Among developers, only 2 had never worked with agile methodologies. 13 developers reported to be knowledgeable in git, 10 had used branching before and 11 used feature branches during the project.
4 developers reported to have significant experience with cloud computing, but 8 had considerable experience in client-server programming. Only 4 developers claimed to have some degree of computer network expertise.
When asked if they would split up microservices into smaller ones or otherwise rearrange the microservices, the majority answered that they would not change it. All others declared that they were not qualified to answer it. Only 3 people thought that the microservices developed reflected Conway's law [36]. 4 people declared to have difficulty remembering which service was called by others. Only 4 people thought that some APIs had poor encapsulation although a total of 9 declared that internal change in the code of called services could break their own. 6 people declared being able to draw an accurate diagram of the service logical architecture.
Only 7 developers had previous docker or virtual machine experience, and only 7 developers built a docker image of their service during the project, showing that the CI automation was considered effective. A mere 3 people declared to be able to draw an accurate infrastructure diagram, while 5 reported to have made significant changes to the IaC code. Of those, all of them rated their own software engineering experience as high or above average and had a computer science background.
In general, the amount of tested code was very low, only 8 declared to have some kind of automated testing. This reflects the general difficulty in testing scientific code reported in the literature. 7 developers declared that their code relied heavily on calls to other services and of those, 4 considered the dependency to be strongly incidental in nature (the other 3 were neutral). A full 9 people declared they would be able to setup everything needed to run their code, which is consistent with the number of people who did not rate the dependency of other services as being high. Only 3 people reported to rely heavily on asking others to make progress their work. 2 people (from the UI team) declared to be blocked in their work by malfunctioning of services on a daily basis. 11 people had participated in debugging activity and valued application logs for this activity. 9 considered debugging to be harder with microservices than monolithic code and of those 5 considered it much harder. 7 debuggers used logging libraries or claimed to have a consistent methodology for print statements. 9 debuggers made use of the log searching tool, and of those 2 had trouble using them or considered them insufficient. To effectively use the log search tool, one had to be able to apply several filters, many based on kubernetes and docker concepts. 8 developers found 3 environments easy to remember and 9 had no problem using 3 branches mapped to these environments consistently. Only 2 declared to have difficulty discovering which version of their code was running in each environment.

Source code metrics
In this section we present source code metrics to provide an overview of the project implementation. Table 1 shows the size in KLOC (thousands of lines of code) and implementation languages for each service. The services called MS-* are RESTFul microservices, while the one called P-* were based on the Kafka message queue. Services were implemented using different programming languages, but python is the most commonly used (12 out of 17 services were either partially or completely implemented in python). The usage of python was not enforced by any technical constraint, so the reason it was chosen for implementation of many services was because many software engineers are proficient in this language. 11 out of 18 (61%) of the survey participants (see Section 4.1) are proficient in python and some of them were involved in the development of two or more services. Furthermore, using the same language share information and even code among software engineers, because many services have similar quality attributes (e.g., logging, handling communication errors, Open API implementation). Microservices are language agnostic because services communicate with each other via lightweight mechanisms [15],REST APIs in the case of this project. Thus, developers can use the best suited programming language for a particular task. For instance, the MS-5, which is heavily based on statistics, has its core implemented in R and the P-7 service was implemented using PROLOG and LISP, which are common in the artificial intelligence domain. The last line accounts for all IaC (infrastructure as a code) implementation, including kubernetes configuration, Dockerfiles and service configuration files. Notably it represents a mere 3% of the overall LoCs, although the CI configuration in the Jenkins is not included.
The size of the service can vary two orders of magnitude, from a few hundreds LOC to dozens of thousands LOC, even when they are implemented using the same language. Some services are inherently more complex than others due to the large number of features provided.

Version control metrics
In order to assess to what extent CI tools have influenced on software engineers, we measured the impact of introducing Jenkins to their daily work routine. We analysed the Git logs of services implementation that started before Jenkins was made available. Jenkins availability started on the 1 st of June 2016 and the Git logs that were analysed since the first commit on 29 th December 2015 until 5 th of September 2018. Based on these Git logs, we selected the commits of 10 software engineers, which were the only that have worked on this project before and after Jenkins availability. We counted the total number of commits per week of these selected software engineers. Then, we divided this total commits per week by the number of software engineers that have committed at least once in the week in order to select only those that were actively working on the project and to remove those on vacation and those that left the project. Three services implementation started before Jenkins availability, namely MS-1, MS-2, and MS-3. Figure 4 shows that the development of two of them was more intense, in terms of lines of code (LOC), before Jenkins was available than after it. Thus, it was not the development intensity that influenced the number of commits. These results are similar to the perceived benefits of DevOps tools observed in other studies, such as Itkonen et al. [37].

Runtime metrics
The prototype had a total of 16 SME users who would test the prototype. Every week day, on average 3 of them accessed the prototype. Over a period of 3 months, we measured a total of over 3 million HTTP requests, of which 72% were completed in under 10ms (as measured by the nginx reverse proxy, not including user network latency). In the same time period, we recorded over 5000 background task execution, the longest of which took almost 7 days, although only 10 took more than 24 hours.

Limitations of study
The most serious threat to validity in this work is that the authors were also developers of the project and also answered the questionnaire. Since the authors account to about 20% of the respondents, we thought that excluding their perspective would impact the validity even more by reducing the sample size and also by excluding a large part of the people with software engineering background.
Another threat to validity in our conclusions is that our microservices evolved together to meet the requirements of a large project, instead of evolving independently as is the norm in the microservices literature. In some cases a same database is accessed by more than one microservice.

Related Work
Idaszak et al. [38] describe the lessons learned when applying DevOps, among other techniques, to a large scientific software project. In this project, a software engineer was appointed as DevOps lead, which allowed SMEs to learn how to use tools like Git and Github, thus simplifying the contributions and increasing collaboration. They concluded that software engineering techniques will foster research by ensuring validity of findings in a long run.
Itkonen et al. [37] conducted interviews with both the supplier organization and their customer to understand how the customer perceived the changes to process and methods, and to identify how the adoption of continuous delivery has had beneficial effects on their work. Their findings were the following: (i) automated testing reduced bugs in production; (ii) development and production environments are similar, which reduced the risk of errors; (iii) simple metrics, such as lead times and release cycle metrics do not give good enough picture of how development works.
Savor et al. [11] analysed data from Git repositories from Facebook and OANDA companies over a period of 7 and 5 years, respectively, in order to investigate the experiences with CI in these companies. Their findings were that CI allows scaling the number of developers and code base size and also the developers prefer short release cycles over long ones.
Subjectively comparing our project to the ones described in this section and also to the ones observed by Segal [1] and Carver et al. [5] and the surveys by Johanson and Hasselbring [2] and Storer [3] we believe that it is representative of scientific computing in general, although there are three important disclaimers to be made: (i) since we do not depend on research grants, our incentive structures are slightly different than in academia in general; (ii) our team does have a large percentage of people with a computer science background; and (iii) compared to the literature we have the advantage of time, since Agile and DevOps already have broad acceptance in the industry and are already past the "peak of inflated expectations" [39].

Conclusion
In scientific information systems development, scientists (SME) and software engineers should work close together to create the multidisciplinary scientific code having high quality of delivery. We presented our practical experience towards bridging the gap in scientific information systems development concerning the use of modern software engineering in scientific computing. We explored the development of a scientific system applying microservices, DevOps, and as-a-service approach delivery from the start to a large applied research project involving a multidisciplinary team of more than 30 people over 3 years.
Our main goal was to validate the hypothesis that the use of microservices and DevOps would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration.
The project was considered successful and has been awarded innovation prizes in both ours and the client's Oil&Gas company. We demonstrated that the adoption of CI and as-a-service delivery has positively contributed to this, based on previous industry projects where we did not apply these techniques. In these other projects, the difficulty of delivering new developments reduced the rate at which client SMEs could test and validate the software, thereby reducing the speed of iterations.
Internally, the CI pipeline also made the overall process easier since people were relieved of the burden of deploying their software manually. In some previous projects, different teams would manually install their software by hand on a shared machine for integration. There was no tracking of what was installed and what were the steps to install it, making any update a risky activity because one could inadvertently break things and not know how to revert to a "sane" state. In contrast, now that almost anything is tracked and reversible, the barrier to test new commits is much lower.
The survey that we conducted also showed that only a small part of the overall team needs to have expert knowledge of CI and DevOps for these methodologies to work.