{'fraction_non_alphanumeric': 0.035513605339605764, 'fraction_numerical': 0.013057507312807214, 'mean_word_length': 4.7276757441418615, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Why are many business instilling a DevOps culture into their organization?

DevOps can be defined as a cultural movement and a technical solution to improve and accelerate the delivery of business value by making the collaboration between development and operations effective, which is rapidly spreading in software industry. However this movement is relatively recent, being necessary more empirical evidence about the real reasons why companies move to DevOps and what results they expect to obtain when adopting DevOps culture. This paper describes empirical research on practicing DevOps through an exploratory multiple case study of 30 multinational software-intensive companies that consists of interviews to relevant stakeholders. This study aims to help practitioners and researchers to better understand the context and the problems that many companies face day to day in their organizations when they do not reach the levels of innovation and software delivery they expect, as well as the main drivers that move these companies to adopting DevOps. This would contribute to strengthening the evidence and support practitioners in making better informed decisions. Furthermore, we have made available the methods to increase the reliability of findings and the instruments used in this study to motivate others to provide similar evidence to help mature DevOps research and practice.


INTRODUCTION
Todays digital landscape demands shorten time-to-market from business ideas are generated until the software supporting these ideas is put into production. High customer expectations forces organizations to adopt an experimental organizational culture through which organizations constantly develop new business ideas and tests these ideas with their customers [4]. This is motivated, and in turn supported, by a prevalent business model based on the web and a software-as-a-service model over cloud infrastructure [4]. Companies that approach this continuous experimentation and that can release software early and frequently have a greater capacity for business innovation, and thus a higher capability to compete in the market. Innovative companies, such as Google, IBM, Microsoft, Amazon, Facebook, LinkedIn, Spotify, and Netflix, are characterized for fast speed in releases and quick response time to customer demands, being able to make multiple deploys per day. These companies have shifted much of the software engineering landscape, mainly organizations culture, skills, and practices [37]. Agile [3] and Lean [38] software development approaches are in the forefront of this shift. However, despite the considerable improvements in software development that these approaches brought [13,41], one of the main problems that slow down speed and agility is the existence of a very strict division of responsibilities between IT departments, which become silos that produce delays in software delivery and decrease software quality. Specifically, we refer to separation in the value chain between those who develop new product features and those who put these features into production.
During 2008-2009 above-mentioned software companies started an organizational transformation to break down these organizational silos and to shift towards fast and frequent delivery of software updates to the customer [18]. DevOps breaks down organizational silos and "stresses empathy and cross-functional collaboration within and between teamsespecially development and IT operationsin order to operate resilient systems and accelerate delivery of changes" [14]. De-vOps requires the development team to work closely and efficiently with the operations team to enable continuous deployment, i.e. to put every change into production through the automation of deployments pipelines, resulting in many production deployments every day [21,32]. Thus, DevOps is an organizational approachalso referred as a cultural movement and a technical solutionthat aims to deploy software updates on continuous basis to production environment whilst also ensure reliable operability of the live environment [34].
Some reports [39,49] prove that DevOps plays, nowadays, a fundamental role for large softwareintensive organizations whose business greatly depends on how efficient development and operations are. However, this movement is relatively recent, and additionally to the cases of these large and well-known companies, it is necessary more empirical evidence about the real reasons why companies move to DevOps and what results they expect to obtain when adopting DevOps culture.
This paper describes empirical research on practicing DevOps in various software development companies. The method is an exploratory multiple case study of 30 multinational softwareintensive companies through interviews to relevant stakeholders. This study aims to help practitioners and researchers to better understand the context and the problems that many companies are facing in their organizations to accelerate innovation and software delivery and the main drivers that move these companies to adopt DevOps, as well as the results they expect to. This would contribute to strengthening the evidence regarding DevOps and support practitioners in making better informed decisions in DevOps transformation processes. There are some decisions that can lead to the failure of an organization, and many others to success, so that the only way to be sure of being on the right way is to follow one that has been successfully proven on numerous occasions. Furthermore, to motivate others to provide similar evidence to help mature DevOps research and practice by replicating the study, we have made available the instruments used in this study, as well as the methods to increase the reliability of findings-such as the inter-coder agreements statistics for thematic analysis.
The structure of the paper is as follows: Section 2 provides an overview of the rising of the DevOps culture. Section 3 describes the method used for the exploratory case study. Section 4 reports the results and findings, whereas Section 5 assesses the statistical validity of these outcomes. Section 6 describes related work. Finally, conclusions and further work are presented in Section 7.

BACKGROUND
DevOps is an organizational transformation that had its origin at the 2008 Agile conference in Toronto, where P. Debois highlighted the need of resolving the conflict between development and operations teams when they have to collaborate to provide quick response time to customer demands [10]. Later, at the OReilly Velocity Conference, two Flickr employees delivered a seminal talk known as "10+ Deploys per Day: Dev and Ops Cooperation at Flickr", which can be considered the starting point to extend agile beyond development [1]. Today an entire industry has been created around DevOps tools whose objective is to automatize best practices, such as continuous delivery and continuous deployment. These practices promote fast and frequent delivery of new and changing features while ensuring quality and non-disruption of the production environment and customers [34].
But beyond all that, DevOps is a cultural movement that aims the collaboration among all the stakeholders involved in the development, deployment and operation of software to deliver a highquality product or service in the shortest possible time. It is a simple concept, but its adoption by organizations is enormously complicated because of great differences in the way in which DevOps promotes to work and the traditional way in which most software companies have been working for decades. As that transformation requires great effort by companies, their CEOs, CIOs, and practitioners in general need evidence about the problems and drivers that are currently moving companies to adopt DevOps. According to the DevOps Agile Skills Association (DASA 1 ) the main drivers are making IT easier, faster and cheaper, and providing more business value. This means, to reduce time-to-market, accelerate innovation, reduce cost, enhance team communication and collaboration, reduce errors, and improve systems stability, among others. Are these findings supported by empirical evidence?

RESEARCH METHODOLOGY: (MULTIPLE) CASE STUDY DESIGN
This paper presents empirical research on practicing DevOps. It is based on the positivism and constructivism models as underlying philosophies [15]. The positivism states that knowledge can be built observing empirical facts and using logical inference; whereas the constructivism or interpretivism states that a phenomenon can be fully understood considering the perspectives of the involved participants and their context. In this research, both approaches are considered useful and a mixed approach is adopted.
The research method is a multiple case study to characterize the reasons why companies move to DevOps and what results they expect to obtain when adopting DevOps culture. This exploratory case study is based on interviews to software practitioners from 30 multinational software-intensive companies from November 2017 to February 2020. The results of the study are presented through an exploratory analysis. Exploratory case studies are useful for finding out what is happening on a phenomenon whilst also seek new insights and generate ideas for new research [33]. The study has been conducted according to the guidelines for conducting case study research in software engineering proposed by Runeson and Hst [43].
The details of the research method was previously described, discussed, and improved at the Fostering More Industry-Academic Research in XP (FIAREX) workshop, part of XP 2018 conference [11], and the Product-Focused Software Process Improvement 2019 conference [12]. We have established a chain of evidence by following a strict process. First, we prepared a questionnaire that was used later for conducting recorded interviews to the involved companies. Afterwards, we transcribed the answers and coded them according to a predefined codebook. Finally, we performed a detailed analysis of the results and their validity. To qualitatively analyze the data, we have used the thematic analysis approach [48,9], which is one of the most used synthesis methods that consists of coding, grouping, interconnecting and obtaining patterns.
3.1. Research Questions. The research questions to be answered through the analysis of this multiple case study can be formulated as follows: • RQ1 What problems do companies try to solve by implementing DevOps? • RQ2 What results try to achieve by implementing DevOps? 3.2. Data Collections and Instruments. The data collection method was semi-structured interviews to software practitioners of 30 companies. The interviews were conducted face-to-face by two researchers from November 2017 to December 2019. The interview consists of almost 100 questions and takes about 2.5 hours.
The preparation of the questionnaire was iterative, hence the questionnaire was first tested with five organizations to evaluate suitability. The questions were collected from existing literature conducting survey studies on DevOps state [39,49,25], exploratory studies [17,34], and meetings with experts in some international and national workshops (the FIAREX workshop [11] and a local industrial workshop organized by the authors 2 ). The questionnaire is structured to collect professional information from interviewees, organizations, context and problems before adopting De-vOps, drivers and expected results, DevOps adoption processes, DevOps teams topology, culture related practices, team related practices, collaboration related practices, sharing related practices, automation related practices, measurement and monitoring related practices, barriers, and results. All questions focus on a unit of analysis, i.e. the department/team in which the interviewee is/was active, and one product/service or project being developed under the DevOps culture and practices.
The questionnaire includes a set of short questions, but also open questions and semi-open questions in which the interviewee can choose one or more options, explain their selections, or add a new option. Both options and questions have been refined as we gained more knowledge during the interviews and workshops. The full interview script and the rest of the case study material is available through the project's web https://blogs.upm.es/devopsinpractice and repositories.
The interviews were conducted face-to-face, using the Spanish language, and the audio was recorded with the permission of the participants, transcribed for the purpose of data analysis, and reviewed by respondents. In the transcripts, each case (i.e. the organization) was given an individual identification number as shown in Table 1.
The interview was promoted through personal contacts of the participating researchers, conferences, professional associations and networks, as well as posts to social media channels (Twitter and LinkedIn).
3.3. Subject Description. We targeted on software-intensive organizations with +2 years of experience in the adoption of DevOps. The sampling for the study can be considered as convenience sampling. The convenience sampling strategy is a non-probability/non-random sampling technique used to create samples as per ease of access to organizations and the relevant stakeholders to the study. Most companies participating in the study were contacted at DevOps related events, such as the DevOps Spain 3 , itSMF events 4 , and DevOpsDays 5 , among others. This may lead to organizations not fully reflecting the entire target audience (e.g. organizations that do not attend to these events), however guarantees that organizations involved in this study are representative of the DevOps movement and its culture. This and other threats to validity are analyzed in Section 3.5.
For each interview, we selected key stakeholders who are members of DevOps teams or know very well the daily work of these teams. If an interviewee did not know to answer a question, that question was answered later after the interviewee obtain that knowledge. Table 1 lists the organizations involved in the study, its ID, scope (international or national), size 6 , business core, organization age, and interviewee IT experience (ages).
3.4. Data Analysis. As this work is primary qualitative research, we have used the thematic analysis approach [48,9]. Thematic analysis is a method for identifying, analyzing, and reporting patterns (themes) within data that are codified segment by segment [9]. Codes are defined as "descriptive labels that are applied to segments of text from each study" [9]. It is convenient to highlight that coding is more than applying codes to segments that exemplify the same theoretical or descriptive idea [20], "coding requires a clear sense of the context in which findings are made" 6 Spanish Law 5/2015 indicates that a micro enterprise is one that has less than ten workers and an annual turnover of less than two million euros or a total asset of less than two million euros; a small company is one that has a maximum of 49 workers and a turnover or total assets of less than ten million euros; and medium-sized companies are those with less than 250 workers and a turnover of less than fifty million euros or an asset of less than 43 million euros; and large companies are those that exceed these parameters. [9]. Themes result from organizing and grouping similar codes into categories that share some characteristic. Themes reduce large amounts of codes into a smaller number of analytic units, and help the researcher elaborate a cognitive map. We used Atlas.ti 8 to instrument the thematic analysis of the interviews [2,22]. According to the use we have made of Atlas.ti, themes are referred to as semantic domains. The method for data analysis we followed is described in three phases: Phase 1. Coding: We applied an integrated approach for thematic analysis [9], which employs both a deductive approach [36] for creating semantic domains and an inductive approach (grounded theory) [6] for creating codes.
First (deductive approach), the Researcher 1 (first author) created a list of of semantic domains in which codes will be grouped inductively. These initial domains integrate concepts known in the literature and discussed in the above-mentioned workshops and events. For domains related to RQ1 (problems), each domain is named P01, P02, P03, etc. For domains related to RQ2 (results), each domain is named R01, R02, R03, etc. Domains were written with uppercase letters (see Figure 1). FIGURE 1. Atlas.ti code manager Second (inductive approach), the Researcher 1 approached the data (i.e. the interviews' transcriptions) with the research questions RQ1 y RQ2 in mind. Researcher 1 reviewed the data line by line (specifically responses to questions Id.11 and Id.12 of the interview), created quotations (segments of text), and assigned them a code (new or previously defined). As more interviews were analyzed the resulting codebook was refined by using a constant comparison method that forced the researcher to go back and forth.
Additionally, the codes were commented to explicitly define the concept they describe, in such a way that they must satisfy two requirements that Atlas.ti defines as follows [2]: exhaustiveness, i.e. the codes of the codebook must cover the variability in the data and no aspect that is relevant for the research question should be left out; and mutual exclusiveness, i.e. (i) codes within each domain need to be different and this needs to be clearly specified in the code definitions, and (ii) at most one of the codes of a semantic domain can be applied to a quotation or to overlapping quotations. This means that the codes should have explicit boundaries so that they are not interchangeable or redundant. We used different colors for each semantic domain and their codes to make the detection of mutual exclusiveness' violations more visible (i.e. no more than one code of the same color can be assigned to a quotation, see Figure 2). The main result of this phase is the list of quotations, which will remain unalterable during the subsequent phases, and the first version of the codebook. The codebook is an important finding of this research and the baseline to answer RQ1 and RQ2. To avoid biases and be confident that the codes mean the same thing to anyone who uses them, it is necessary to build that confidence. According to Krippendorff, reliability grounds this confidence empirically [28] and offers the certainty that research findings can be reproduced [2]: "The more unreliable the data, the less likely it is that researchers can draw valid conclusions from the data" [2]. Phase 2. Improving reliability of coding. We used Inter-Coder Agreement (ICA) analysis techniques for testing the reliability of the obtained codebook. This is an iterative process in which Researcher 2 (second author) worked as Coder 1 and Researcher3 (third author) worked as Coder 2. Coder 1 and Coder 2 were involved to coding the interviews' transcriptions by using a codebook, which was iteratively refined while the ICA did not reach an acceptable level of reliability. This phase has to be very methodical, so Researcher 1 provided to Coder 1 and Coder 2 with detailed procedures about how to use codes and Atlas.ti projects (e.g. if codes within a semantic domain are not applied in a mutually exclusive manner, the ICA coefficient is inflated and cannot be calculated). After coding, Researcher 4 (fourth author) calculated and interpreted the ICA between Coder 1 and Coder 2. If coders did not reach an acceptable level of reliability, Researcher 1 analyzed the disagreements to find out why Coder 1 and Coder 2 had not understood a code in the same manner, and delivered a new version of the codebook. Next, Coder 1 and Coder 2 made a new coding on a new subset of interviews' transcriptions. This process was repeated until the ICA reached an acceptable level of reliability. This is one of the most important phases for the validity of the study and also, one of the most complex, because, to our knowledge, there is no a complete support for ICA statistics in qualitative analysis tools. To deal with this gap, Researcher 4 established a novel theoretical framework for measuring and evaluating the ICA based on Krippendorff α-coefficient and coefficients implemented by Atlas.ti. Section 3.5 describes the validity procedure in detail and Section 5 its execution.
Phase 3. Synthesis. The final product of this step can be a description of higher-order themes, a taxonomy, a model, or a theory. The first action is to determine how many times each domain appears in the data in order to estimate its relevance (grounded). The second action is to support the analysis with evidence through quotations from the interviews. The third action is to calculate the co-occurrence table between units observed, this means, between problems' domains, and between results' domains. The fourth action is to create semantic networks, i.e. analyze relationships between domains (association, causality, etc.) as well as the relationship strength based on cooccurrence. These relationships determine the density of the domains, i.e. the number of domains you have related to each domain. It is possible to repeat these actions for each code within a domain. We did it for more grounded codes. Finally, it is possible to quantitatively analyze the problems and results by case (organization) or analyze the relationships between problems and results-i.e. interconnecting categories.
3.5. Validation procedure. We have followed the strategies pointed out by Creswell [7] to improve the validity of exploratory case studies, as follows.
1. Data triangulation so that the data of this study are gathered from a number of companies that is enough to build a complete picture of the phenomenon. At this regard, Crouch & McKenzie [8] propose that less than 20 participants in a qualitative study helps a researcher build and maintain a close relationship and thus improve the open and frank exchange of information. This can help mitigate some of the bias and validity threats inherent in qualitative research. Consequently, the sweet spot sample size for many qualitative research studies is 15 to 20 homogeneous interview participants. In this research, 30 companies participated and +30 relevant stakeholders were interviewed. This multiplicity is what provides the basis for generalization [42].
2. Member checking so that the participants received the transcribed interview and the preliminary results to ensure the correctness of our findings.
3. Rich description so that the context of the involved organizations/teams are described as far as confidentiality issues allow.
4. Clarify bias, i.e. those related with the qualitative research method like the bias of the authors and those related with the participating organizations like their location. The first one was mitigated using Inter-Coder Agreement (ICA) in the thematic analysis, and the second one was mitigated by the diversity of the interviewees (organizations from different business and industries, in different countries, and from different stakeholders and roles), which increases the generalizability of our results.
5. Report discrepant information so that all the results have been presented and analyzed, regardless of its implications for our initial interests. Prolonged contact with participants, the duration of the interviews, and the subsequent communication allowed us to fully understand their perspectives.
From all these strategies, it is especially relevant the method used in this paper to reduce authors bias. As mentioned before, to evaluate the reliability and consistency of the codebook on which the study findings are based, we applied ICA analysis techniques. ICA analysis is a toolbox of widely used statistical techniques that provide a formal and standardized way of quantifying the degree of agreement that several judges achieve when evaluating a certain amount of raw material.
There exists in the literature a variety of measures of ICA (c.f. [5,35,19,44,47]) that may be applied to different situations. However, for our purposes, we focus on Krippendorff's α coefficient [23,26,27,29]. This measure is a standard tool for quantifying the agreement in content and thematic analysis due to its well-established mathematical properties and probabilistic interpretations. In this research we have used the following Krippendorffs α coefficients: • The coefficient α binary : This coefficient is computed on a specific semantic domain S. It is a measure of the degree of agreement that the judges achieve when choosing to apply a semantic domain S or not. • The coefficient cu-α: This coefficient is computed on a semantic domain S. It indicates the degree of agreement to which coders identify codes within S. • The coefficient Cu-α: In contrast with the previous coefficients, this is a global measure of the goodness of the partition into semantic domains. Cu-α measures the degree of reliability in the decision of applying the different semantic domains, independently of the chosen code. Figure 3 shows an illustrative example of the use of these coefficients. Let three semantic domains and their respective codes be: Coder 1 and Coder 2 assign codes to four quotations as shown in Figure 3(a). Each coder and each code are represented as shown in Figure 3  The coefficient α binary is calculated per domain (i.e. S 1 red, S 2 blue, S 3 green) and analyzes if the coders assigned or not a domain-whatever the code-to the quotations (see Figure 3(c)). Notice that α binary = 1 for S 2 as both coders assigned this domain to the second quotation and the absence of this domain in the rest of quotations, i.e. total agreement.
The coefficient cu-α is calculated per domain (i.e. S 1 red, S 2 blue, S 3 green) and analyzes, given a domain S i , whether the coders assigned the same codes to the quotations (see Figure 3(d)). Notice cu-α = 1 for S 2 as both coders assigned the same code to the second quotation and no code from this domain to the rest of quotations, i.e. total agreement. Also notice that cu-α < 1 for S 3 as the coders assigned the same code of S 3 to the third quotation 3 but they did not assigned the same codes of S 3 to the rest of quotations. Finally, observe that cu-alpha for S 1 is very small (near to zero) since the coders achieve no agreement on the chosen codes.
Analogously, the coefficient Cu-α analyzes all the domains but it does not take into account the codes within each domain (see Figure 3(e)). Notice that Cu-α < 1 as both coders assigned the same domain S 1 to the first quotations and the domains S 1 and S 3 to the third quotation, but they did not assigned the same domains in the second and fourth quotations.
The bigger the coefficients α are, the better agreement is observed. A common rule-of-thumb in the literature [28] is that α ≥ 0.667 is the minimal threshold required for drawing conclusions from the data. For α ≥ 0.80, we can consider that there exists statistical evidence of reliability in the evaluations.
In Appendix A we provide a detailed description of these coefficients with the aim of filling the gap between the multiplicity of descriptions presented in the literature (sometimes too vague) and the current implementation at Atlas.ti (for which only a very brief description is provided in the user manual). This Appendix describes a novel theoretical framework that allows us to reinterpret the variants of the α coefficient as different incarnations of the same reliability measure. This measure is computed as the rate of observed disagreement and expected disagreement, following Krippendorff (see Appendix B for a detailed mathematical formulation). The combined methods of these two Appendices gives rise to a new simple and precise algorithm of computation of these coefficients, as well as the semantics provided above.

(MULTIPLE) CASE STUDY RESULTS
This section describes the analysis and interpretation of the results for each research question: RQ1 and RQ2. The authors were involved in two iterations of validation and improvement of the codebook related to the problems that companies try to solve by implementing DevOps and results that companies try to achieve by implementing DevOps, until an acceptable ICA was reached (α ≥ 0.80) (see Section 5). A complete version of this codebook in which each code is described in detail is available at GitHub 7 . 4.1. RQ1: What problems do companies try to solve by implementing DevOps? Table 2 shows the codebook version 2 that lists 10 semantic domains and 35 codes related to the problems that promote to companies to adopt a DevOps culture. Table 2 also shows how many times each domain and each code appears (i.e. code usage frequency) in the interviews' data in order to estimate their relevance (grounded). Hence, it is possible to check that the P01 "Too much time for releasing" is the most mentioned problem by the companies participating in the study (grounded = 27). Behind, P10 "Digital transformation drivers" follows (grounded = 15), understood as (i) agile and lean drivers; (ii) movement to DevOps due to clients demands, market trends and hypes; and (iii) the need to initiate a transformation due to technological obsolescence or large architectural, infrastructural, and organizational changes. Some other recurring problems are P09 "Lack of standardization and automation" (grounded = 14), P02 "Problems when releasing new versions" (grounded = 13), P06 "Organizational and culture silos" (grounded = 10), P03 "Too much time spent on setting up environments" (grounded = 9) and P07 "Lack of collaboration between Dev & Ops" (grounded = 9). Finally, Figure 4 shows a fragment of this codebook, as reported by Atlas.ti, with explanatory comments of codes.  The relevance can also be measured by the total of words that a semantic domain accumulated in the interviews' transcriptions. The analysis provided by Atlas.ti related to the total of words also supports that P01, P10, and P02, in this strict order, were the most relevant, whereas P04, P05, and P08 were the least relevant.
Next, we analyze the most relevant codes-based on their grounded-, which show the problems that instilled a DevOps culture, and we provide some evidence for each code through excerpts of the interviews' transcriptions: P01a "Need of being more agile, rapid", is the most relevant code, which is mentioned up to 23 times (grounded = 23). Hence, the organizations claimed that the most important problem is the too much time for releasing new features into production. They need to reduce time-to-market and be more agile/rapid to adapt to market needs and demands, either new features or updates. Business demands more velocity than teams can offer. Hence, the organizations emphasized the too much time deploying, delivering, and releasing new features and hotfixes, and talked about the need of accelerating the "value" delivery to customers. P01a is evidenced by the following excerpts: "The time between development teams finished a new software and it was deployed into production was very long, sometime months" [ID01], "The main concern is the time it takes to put a release into production. Without emphasizing the origin of the problem, the client told us that it takes a long time to put into production the requested changes, and that they want to adopt DevOps to reduce this time" [ID06], "The speed demanded by business was not according to the speed provided by operations. It took us a long time to develop features and when they went into production, there was something else new. This way of working does not allow us to iterate and adapt to market needs quickly" [ID11], "The client had very traditional developments based on waterfall methodologies and it was very difficult for them to respond to the business needs. Every time they wanted to make a change and put that change into production it took about 3 or 4 months" [ID19], "We were late, frequently, we could not deploy into production because software applications were not working or hotfixes were being solved. When we solved an operating error, four more errors appeared later" [ID16], finally "The most important driver is that we were unable to generate a hotfix quickly and over-the-air" [ID22].
P10b "Business/market demands or trends" (grounded = 7), which is related to digital transformation drivers above-mentioned. This indicates that many times the reasons for adopting DevOps do not come from within the organization, but from outside, such as customer and market requests. P10b is evidenced by the following excerpts: "We started adopting DevOps because of customer demands" [ID08], "The clients we work with were starting a DevOps transformation because the trend that is being sold from the market is that with the introduction of DevOps you are going to get a higher quality software in less time. The trend has a lot of influence" [ID16], "We haven't adopted DevOps as a resolution to a problem, but as a need to adapt to a market with enormous variability that requires much more agility than we were capable of providing" [ID04], finally "The market is asking for DevOps. We came from applying traditional development models and were not aware of the market situation and technological evolution. We had to make a change within the organization and adapt us to a new model" [ID14].
P02a "Need of higher quality (released) products/services" (grounded = 7). This means that the organizations participating in the study need velocity, but they also need high quality products and services, i.e. they need to release software with the least number of bugs as possible, improve user experience, and other quality metrics, such as performance. P02a is evidenced by the following excerpts: "In the end the problem was faster value delivery, but also higher quality delivery and quicker problem solving" [ID30], "Production deployments needed a lot of speed and quality" [ID09], finally "The performance of the service was very limited, from a certain number of users connected, navigation on the platform became slow and unstable. We needed to improve the user experience (browsing speed, minimize performance drops,...) and optimize hardware resources" P03d "Too much time on configuring environments" (grounded = 5). It is striking that large and important companies have such important problems when configuring their environments. P03d is evidenced by the following excerpts: "Setting up new environments or updating existing environments was very time consuming. In addition, different teams shared environments so it was very difficult to agree on the changes that affected the environments in which the different developments were deployed" [ID01]. "You asked for some machines and the IT department took two months; you never knew when your request would be resolved (maybe there were thousands of requests in such a large company). After two months the machines were delivered, but without IP, and you entered a list of requests again to request the IP. We weren't well organized and each team was very slow" [ID02].
P07b "Problems/lack of communication" (grounded = 5) and lack of understanding between development and operations. Software products and services are built or provided through interaction and collaboration of very diverse professionals. If these professionals do not communicate adequately the timing and quality of the product/service will be compromised. P07b is evidenced by the following excerpts: "There was a lack of communication between the operations and development engineers, which meant that they did not understand each other" [ID01], "One of the problems we encountered was lack of communication between the developers and deployment teams. The deployment team was completely unaware of what was being implemented" [ID12], "There was also a lot of distance, in terms of understanding, between what people developed and wanted to deploy and how they deployed. There was not much communication with operations and they were not even prepared to meet the deployment requirements" [ID11], finally "In the end, the lack of collaboration between development and operations caused many delivery failures" [ID10].
P06a "Biz & Dev & Ops have different goals (business or functional requirements)" (grounded = 4), which discloses the problem caused by the goals misalignment of the parties involved in the delivery of a software product/service, i.e. business, development and operations. Hence, operations personnel do not inform about operational requirements or changes to developers, whereas operations do not receive sufficient information about how systems work and do not have sufficient understanding of the business. Some organizations participating in the study claimed that "There were barriers between development and operations teams as they did not share common objectives. Development was focused on implementing new functionality and operations in maintaining the system stable" [ID26], "On the one hand, developers want to deploy more often, because they want to bring their software into production. On the other hand, system engineers are looking for application stability, and changes destabilize" [ID01], and "Development worked very close to business, but very far from operations. So, the software functionally fulfilled most of the requirements of the business, but the software did not meet the operational requirements. The system sometimes went down because millions of people were simultaneously accessing to it and no one had thought about it" [ID25].
P09b "Lack of processes automation" (grounded = 4) and P09e "Need of automating infrastructure creation and configuration" (grounded = 3), which are related to the need of automated processes for building, testing, integration, deploying, and releasing software, and the need of automating infrastructure (infrastructure as code, immutable infrastructure, etc.). Some organizations revealed that manual deployments generate problems and failures, and have associated a high cost. These codes are evidenced by the following excerpts: "The manual deployments generated failures from time to time; in fact, there were some manual errors that generated important problems" [ID22], "Millions of euros were spent on deployments, which were manual" [ID25], "The lack of automation in the processes slowed down a lot the releasing time." [ID10], "We are going to be a company 24/7 and we need that releases do not stop production, so we need a quality deployment pipeline with a lot of automation and control" [ID09], and "We needed to have automation of the infrastructure of the environments, dynamic growth of it, dynamic provisioning, and thats when we saw that we needed DevOps" [ID05].
Finally, it is also noteworthy that the larger the companies are, the bigger the organizational problems are. Hence, some organizations participating in the study reveal that "This organization is very large and matrix. Development and infrastructure were silos that make successful release very difficult" [ID02]. "We realized that the very size of the company, the diversity of departments (development, operations, security, service, QA, architecture, etc.), and the complexity of the processes, as well as the interaction between them, went against reduce time-to-market, and made us less competitive" [ID17]. These excerpts are related to P06d "Organizational silos", although there also exist P06c "Information/knowledge silos" that do not explicitly respond to organizational silos like departments. Additionally, some organizations revealed that operations are not sized for assisting development requirements (P03a): "The operations personnel were overwhelmed by all the resources they had to provide to development teams so that they couldn't improve their tasks. It is primarily the bottleneck in the software development process" [ID13]. 4.1.2. Semantic Networks. The above-mentioned domains and codes are related to each other through semantic networks as described next. Figure 5 shows a semantic network about the problems that are stilling to organizations to move to DevOps and how these problems are related to each other. An analysis of co-occurrences among the semantic domains is used to establish relations between domains and to know how strong are these relations (see Table 3). Each cell of this table shows the co-occurrence of two semantic domains in a quotation. In the network, these values determine the width of the line connecting the domains, i.e. the higher the co-occurrence, the thicker the line. Furthermore, the network establishes the type of the relations between two domains. Two types of relationships are identified: is associated with, which is generic, and is cause of, which indicates causality (but in a deductive, not statistical, sense). Finally, the size of the boxes (specifically the value of the height) containing the semantic domains indicates their grounded value, i.e. the higher the grounded, the taller the box. These considerations must be taken to properly understand the semantic network. Next, we analyze the network from the most to less grounded domains as follows: The problem P01 "Too much time for releasing" (grounded = 27) has a strong relationship with P02 "Problems when releasing new versions" (co-occurrence = 9). It can be deduced that releasing problems cause that delivering software takes too much time. P01 has also a strong association with P10 (co-occurrence = 5), hence it seems that the digital transformation drivers of the companies impact in their time-to-market, but we could not determine causality reasons. Finally, P01 is related to P07 "Lack of collaboration between dev & ops", P09 "Lack of standardization and/or automation", P06 "Organizational/cultural silos", and P03 "Too much time spent on setting up environments", likely causing the excessive time that companies invest in releasing software.
The problem P10 "Digital transformation drivers" (grounded = 15) is associated with P02 and P09 (also P01 that was above mentioned). Thus, it seems that the digital transformation drivers are related to the lack of standardized and automated processes, technological stacks, infrastructure, etc., as well as the problems when they releasing software.
The problem P09 "Lack of standardization and automation" (grounded = 14) is associated with P06 and P07, i.e. the silos, either organizational or cultural silos, and the lack of collaboration between development and operations teams. It can be deduced that the lack of collaboration makes it difficult or even impossible the standardization of methods, tools, and technologies, as well as the automation of processes.
The problem P02 "Problems when releasing new versions" (grounded = 13) is related to P01 and P10, already mentioned, but also to P06 and P07. Thus, it can be deduced that organizational and cultural silos, as well as the lack of collaboration between dev and ops, cause problems when releasing new versions.
The problem P06 "Organization and cultural silos" (grounded = 10) has a strong relationship with P07, in such a way that the lack of collaboration between Dev and Ops causes these organization and cultural silos in the companies participating in the study. In turn, organization and cultural silos cause systems' downtimes.
Finally, P03 "Too much time spent on setting up environments" (grounded = 9) is caused by P07 and P09, i.e. by the lack of collaboration and the lack of standardization and automation, respectively. 8 Note that, for simplicity, we represent the occurrence relation only in the upper-diagonal part Additionally to the semantic network at the domain level, Figure 6 shows a semantic network at the level of codes, specifically around the most grounded, and thus, outstanding code. This network is based on the co-occurrences between this code and others (see Figure 7). As said before, the most grounded code is P01a "Need of being more agile, rapid" (grounded = 23) and this is the base code of this network. The network is created following the guidelines above-mentioned regarding the lines and boxes. The network relates the base code with other elements through three kind of relationships: is part of, which links the code to its domain, is associated with, which is generic, and due to, which indicates causality (but in a deductive, not-statistical, sense).
P01a has a strong relationship with P02a "Need of higher quality products" and P10b "Business and market demands or trends" (co-occurrence = 5). This means that the organizations that mentioned P01a in a quotation, also mentioned these codes, so agility, quality and business/market demands and trends are intrinsically linked. P01 has a medium relationship with P06d "Organizational silos", P02c "Releasing problems", and P07b "Lack of communication" (co-occurrence = 2). From this it can be deduced that the need of being more agile may, to some extent, is due to these problems. Finally, P01a is weakly related to P03d "Too much time on configuring environments", P09a "Complex processes", and P02b "Need of higher quality deployments" (co-occurrence = 1).   Table 4 shows the codebook that lists 6 semantic domains and 18 codes related to the results and benefits that companies expect to achieve as a result of adopting DevOps. Table 4 also shows how many times each domain and each code appears (i.e. code usage frequency) in the interviews' data in order to estimate their relevance (grounded). Hence, it is possible to check that the R01 "Faster time-to-market" is the most mentioned result by the companies participating in the study (grounded = 27); thus, it seems to be the most important result that organizations expect when initiate the adoption of the DevOps culture. This result is followed by other relevant ones like R03 "Improve process productivity" (grounded = 20), R02 "Better software quality" (grounded = 19), and R04 "Improve team effectiveness & satisfaction" (grounded = 13). Other expected results expressed to a lesser extent by the companies are the R05 "Customer focus" (grounded = 7), and the R06 "Align the objectives of Dev & Ops with Business" (grounded = 2).
The relevance can also be measured by the total of words that a semantic domain accumulated in the interviews' transcriptions. The analysis provided by Atlas.ti related to the total of words also supports that R03, R01, R04, and R02 in this strict order, were the most relevant, whereas R05 and R06 were the least relevant (see the study repositories).
Next, we analyze the most relevance codes-based on their grounded-, which show the results that organizations expect to achieve when adopting a DevOps culture, and we provide some evidence for each code through excerpts of the interviews' transcriptions: R01d "Faster time-to-market" is the most relevant code, which is mentioned up to 19 times (grounded = 19). Hence, the organizations reported that the most expected benefit when they initiated their DevOps transformation was to accelerate time-to-market by reducing development, testing, quality assurance, deployment and delivery times. R01d is evidenced by the following excerpts: "It was expected that DevOps enables the development of new functionalities and their deployment into production for final users more quickly" [ID21], "We expected that DevOps allows us to give a faster response to the client. Nowadays agility is an absolute necessity" [ID04], and "By incorporating DevOps we will gain agility and achieve a better adaptation to the current changing market" [ID14].
R02a "Better software quality" (grounded = 15). The organizations also reported important expected results and benefits related with the increase of software quality. These results are supported by the following excerpts: "We expected an improvement on the quality of the deliverables" [ID14], "It was expected that DevOps technologies support us to deliver software faster, but also to build software with better quality. In the end, you have thousands and thousands of automated tests that are going to be executed for new code that you want to deploy" [ID03], and "We are looking for productivity and efficiency as well as quality in software. These results facilitate what top management wants: fast time-to-market and competitiveness" [ID05].
R03a "Process automation: efficiency, optimization, productivity" (grounded = 15). R03a is evidenced by the following excerpts: "We will be able to deploy at any time" [ID22], "We aim an automated model in the provisioning of the platform" [ID27] and "We expected the benefits of automating tasks, previously done manually, through continuous integration and continuous deployment. This avoids human mistakes and shortens time to production" [ID24]. In this regard, some striking issues that organizations expect to eliminate by adopting DevOps are described through the following excerpt: "There were several manual changes made by particular developers and this caused several problems. Sometimes these developers were on vacation and no one knows what they touched. Sometimes, a change was made by hand in production and the next time something was uploaded and overwritten, the change was lost and no one remembered how to do it" R04c "Improve team collaboration and communication" (grounded = 8). R04c is evidenced by the following excerpts: "The aim of incorporating DevOps was to improve communication and dialogue between all parties involved in the projects" [ID12], "We were looking for breaking down all the barriers that initially existed between the development and operations teams so that everyone share common objectives"[ID26], "We were looking for a satisfactory process for development and operations to generate more confidence. Operations did not trust on that the developers were able to deploy in production, meanwhile developers were always criticizing that they were not allowed to deploy" [ID01], and "The main result we were expecting was to reduce the distance between the development and operations teams. We do not want to spend time on deploying code, integrating code, solving problems in different branches, etc. We want to spend time on design, development, learning about new technologies, innovation, etc. This makes the company much more competitive" [ID24].
Finally, the organizations also highlight that they expect that software products are more reliable and resilient (R02b), the management of the projects is more effective (R03b), and the IT cost is reduced (R03c). This can be observed in excerpts like these: "We expected an improvement in response times to incidents, performance and communication of the systems that support the platform, as well as more effective project management" [ID07], "As a result of changing traditional project management to product-oriented management, we expected to improve the management and shorten development, deployment and QA times" [ID12], "We wanted to bring the operations teams closer to the development and business teams. That is to say, if business asked for a new website, our idea was to give an end-to-end vision of this product. We did not want development and operations working separately, we wanted everything to be a single stream leading up to a new website" [ID02], and "In the end, the main role of management is to seek fast time to market and decrease IT costs" [ID5].
Once the most grounded codes were analyzed-and in turn, the most relevant domains R01-R04-, we focus on the codes framed in R05 "Customer focus" (grounded = 7). The companies also reported some expected results related with customer-centered approaches. The organizations claim that by introducing DevOps they expect to manage customers response faster, deliver greater value, obtain feedback more quickly and continuously, and innovate more. These expected outcomes can be found in the following excerpts: "Above all, we expect providing greater value to customers and getting higher satisfaction" [ID08], "Improve the user experience, faster feedback from the end user so that the business gets that feedback" [ID11], "By making cycles faster we aims to get better customer feedback and expect that customers get what they really want, and that the user experience is better too" [ID10], finally 'DevOps makes innovation less scary. We expect to spend more time on new things because pipelines breaks less so that you do not have the experience that production has broken down for several days because you have tried something new" [ID03].

Semantic Networks.
The above-mentioned domains and codes are related to each other through semantic networks as described next. Figure 8 depicts a network about the results that organizations expected to achieve when they initiated the DevOps adoption and how these results are related to each other. Table 5 presents the co-occurrences analysis at domain level that enables the creation of the network. These networks are created following the guidelines previously mentioned: the height of the boxes is determined by the grounded value of the elements, the relations between them are based on the analysis of co-occurrences, the co-occurrence values determine the line width that relates the elements, and several types of relations characterize these relationships. Next, we analyze the network from the most to less grounded domains as follows: 9 Note that, for simplicity, we represent the occurrence relation only in the upper-diagonal part The expected result R01 "Faster time-to-market" (grounded = 27) has also the largest number of relationships and these relations are also the strongest. Therefore, it could be inferred that this is the most relevant result that companies intend to achieve when adopting DevOps. R01 has a strong relationship with R03 "Improve process productivity" (co-occurrence = 10). It can be deduced that the optimization and automation of processes causes faster time-to-market. R01 is also strongly associated with R02 "Better software quality" (co-occurrence = 7), confirming the dilemma between speed and quality. R01 is also related to R04 "Improve team effectiveness & satisfaction" (co-occurrence = 4), in such a way that not only is important the effectiveness of process, but also people satisfaction and team effectiveness. Finally, R01 is weakly related to R05 "Customer focus" (co-occurrence = 1) and R06 "Align the objectives of Dev & Ops with Biz" (co-occurrence = 1).
The expected result R03 "Improve process productivity" (grounded = 20) is associated with R04 (co-occurrence = 3) and R05 (co-occurrence = 1), indicating that processes, people & teams, and customers are intrinsically related. R03 is also strongly related to R02, so it can be deduced that the improvement of processes productivity may lead to better software quality.
The expected result R02 "Better software quality" (grounded = 19) is strongly related to the semantic domains R01 (co-occurrence = 7), R03 (co-occurrence = 4) and R04 (co-occurrence = 3), and weakly related to R06 (co-occurrence = 1). In a nutshell, it can be deduced that the adoption of the DevOps culture in software companies can favor higher levels of process productivity and team effectiveness and satisfaction, and in turn, it may lead to improve software quality.
Finally, the expected result R04 "Improve team effectiveness & satisfaction" is related to the above-mentioned R01, R02, R03, and associated with R05, in such a way that the improvement of team effectiveness and people satisfaction is related with the improvement of customer satisfaction and experience, by providing an excellent service to customers.
Additionally to the semantic network at the domain level, Figure 9 shows a semantic network at the level of codes, specifically around the most grounded, and thus, outstanding code. This network is based on the co-occurrences between this code and others (see Figure 10). As said before, the most grounded code is R01d "Faster time-to-market" (grounded = 19) and this is the base code of this network. The network is created following the guidelines above-mentioned regarding the lines and boxes. The network relates the base code with other elements through three kind of relationships: is part of, which links the code to its domain, is associated with, which is generic, and due to, which indicates causality (but in a deductive, not-statistical, sense).
R01d has a strong co-occurrence with R03a "Process automation: efficiency, optimization, productivity" (co-occurrence = 7), which is also very relevant (grounded = 15). This means that the organizations that mentioned R01d in a quotation, also mentioned R03a. Thus, it can be deduced that faster time-to-market is largely due to processes automation. R01d is also strongly associated with R02a "Better software quality" (co-occurrence = 4), confirming the dilemma between fast speed and high software quality. R01d is also related to R04c "Improve team collaboration & communication" (co-occurrence = 2), and henceforth, it could be understood that the increase of velocity may be also due to the improvement of team collaboration. Other codes weakly related to R01d are R04d "More motivated teams", R06a "Align objectives with business" and R03b "Project management more effective" (co-occurrence = 1). The faster time-to-market that companies aim to achieve could be also due to these expected results.  Figures 5 and 6 we can deduce some common behaviour patterns as follows: • The relationships between the domains P01, P02, and P10 ( Figure 5) show that organizations, due to market demands and trends, are forced to shorter releasing cycles; however, the effort invested on high quality software products and services may delay releasing new features. This means that the organizations participating in the study may be not prepared to deal with the dilemma between speed and quality. This can be also checked at the code level through the relationships between P01a, P02a, and P10b ( Figure 6). Hence, market needs (P10b) implies that organizations need to be more agile and fast (P01a), but at the same time they are forced to release new high quality products (P02a). This pattern is not new, but it is novel to see that organizations really think that DevOps is a solution to this dilemma.
• The relationships between the domains P01, P06, and P07 ( Figure 5) demonstrate that organizations need a cultural change. This means, the data shows that the lack of collaboration between development and operations teams generates silos, both cultural and organizational silos, although we could also deduce that the silos generate lack of collaboration. What is clear is that organizations identify the silos and the lack of collaboration as a cause of the too much time spent in releasing new features (Figures 5, relationships P06-P01 and P07-P01) • The great number of problems associated with P01a "Need of being more agile, rapid" is significant (Figure 6). This is an indication that organizations think that DevOps is a possible solution to (i) reduce time-to-market, and (ii) adapt to business needs and market demands. • The problem P05 "Barriers to innovation and experimentation" has a low relevance (grounded = 2) and is not mentioned or related to other domains (see Tables 2 and 3, respectively). The fact that organizations have not brought up issues like the flexibility and autonomy of the teams to enable and promote innovation and experimentation is relevant, as other sources of literature (e.g. DASA) do. This means that either (i) organizations do not experience barriers to innovation, or (ii) organizations experience these barriers but they do not associate this problem with the reasons that instilling to organizations to adopt a De-vOps culture. Our impression after analyzing all the data is that organizations initiated a DevOps transformation with the immediate objective of dealing with the too much time spent on releasing features in order to satisfy their customers-which is their most important problem-, but the capacity of rapid innovation and experimentation is a great goal to be dealt with in a second wave. • The problem P08 "Lack of end-to-end vision of value stream" has also a low relevance (grounded = 2) and is not mentioned or related to other problems (see Tables 2 and 3, respectively). Developers have too little focus on production environments and how systems work in these environments, whereas operations are involved in the development too late. The problems about the lack of responsibility by developers after they have been put systems into production-non-shared end-to-end responsibility-, and the lower level of ownership that operations personnel possess with respect to developers, were also revealed, although with less emphasis. • The problem P04 "System downtime" is also rarely mentioned and rarely associated with other problems (see Tables 2 and 3, respectively). This could be because most organizations participating in the study have values for mean-time-to-recover (MTTR) that are suitable for their respective business domains.

4.3.2.
About results. From the networks of Figures 8 and 9 we can deduce some common behaviour patterns as follows: • The causal relationships between the domains R01, R03, and R04 ( Figure 8) demonstrate that organizations expect that, by improving processes and team efficiency, time to market may be accelerated. This hypothesis is confirmed by the relationships between the codes R03b, R03a, R04c, R04d and the code R01d (Figure 9). • The causal relationships between the domains R02, R03 and R04 (Figure 8) show that organizations expect that, by improving process and team efficiency, they may generate higher quality software.
• The association between the domains R01 and R02 (Figure 8) shows that organizations expect that the goal of reducing time-to-market is not achieved at the expense of product quality. Specifically, the relationships of R01d with R02a (better software quality) and R02b (reliability, resilience, availability) highlight this point.

4.3.3.
About problems and results. Figure 11 shows that, as a general rule, organizations identify the problems they have and that motivate them to adopt DevOps better than the expected results they can achieve. This demonstrates the general lack of knowledge about the promising benefits of DevOps culture, maybe motivated by the lack of empirical evidence. FIGURE 11. Relation between the identification of problems and expected results by organizations (in words) Figure 12 depicts a semantic network that shows the existing relationships between the problems that lead an organization to adopt DevOps and the results they expect to obtain from such adoption. The network establishes four types of relations: is associated with, due to, is part of, and matches. This network is described as follows: • The expected result R01 "Faster time-to-market" is a mirror of the problem P01 "Too much time to for releasing", which in turn is associated with P02 "Problems when releasing new versions", as is shown in Figure 13 by querying data in Atlas.ti. Figure 13 shows the codedocument table for the query P01 AND P02 AND R01. Although the table does not show the 30 cases, the resulting co-occurrence is 9. • The problem P01 is due to the problems P06 "Organizational and cultural silos" and P07 "Lack of collaboration between Dev & Ops" and the expected results when these problems are addressed are those embodied in R04 "Improve team effectiveness and satisfaction". • The problem P01 is due to the problems P03 "Too much time spent on setting up environments" and P09 "Lack of standardization and automation" and the expected results when these problems are addressed are those embodied in R03 "Improve process productivity". • The problem P02 is mainly due to the problems P06 and P07. The organizational and cultural silos, and specifically the problem described by the code P06c "Information and knowledge silos", match R04c "Improve team collaboration & communication". • The problem P02a "Need of higher quality products and services" matches R02a "Better software quality" and R02b "Reliability, resilience, availability".

4.3.4.
DevOps Adoption Patterns and Anti-Patterns. Patterns are described as follows: • Time-to-market rusher. Reducing time to market and responding to market demands, either for new features or updates, are the two main drivers why companies are instilling  Market forces organizations to be more agile and faster, but at the same time they are also forced to releasing high quality software products. Organizations that start a DevOps transformation expect that reducing time to market is not at the expense of product quality. Indeed, they also expect to improve quality by introducing DevOps. • Silos remover. The existence of organizational and cultural silos (e.g. matrix structures) causes the lack of collaboration between the development and operations departments, and in turn, this lack of collaboration generates knowledge silos. The existence of these silos and the lack of collaboration cause delays in releasing new versions. Organizations that start a DevOps transformation assume that the adoption of DevOps will improve the effectiveness and efficiency of (i) teams, i.e. better communicated and collaborative, and more motivated teams, (ii) process, due to the automation of continuous integration and deployment pipelines, and (iii) project management. • Efficiency enabler. Organizations that start a DevOps transformation expect that if they improve the efficiency of teams and processes (e.g. automation, standardization of technology stacks, etc.), this will result in higher quality of the final product.
Anti-patterns are described as follows: • Snowball-er. Some organizations start a DevOps transformation as consequence of a large architectural (e.g. microservices) or infrastructural (e.g. cloud) change. Since they have to change, they also adopt DevOps, usually conceiving DevOps as a processes automation and ignoring any cultural and organizational approach. • Headless chicken. Some organizations start a DevOps transformation for being a trend or hype, or in the case of consulting organizations due to their clients ask for it, but they do not analyze cost, benefits, and risks.

VALIDITY
This section describes the ICA analysis to assess the validity of our study. For this purpose, we report the values of the several variants of Krippendoff's α coefficient achieved during the codification process. A precise definition and interpretation of these measures is summarized in Section 3.5 and Appendix A.

5.1.
Results for α binary . In our case of study, the results for the α binary coefficient for the research question RQ1 (problems) on the semantic domains defined in Section 4.1, for the two performed iterations, are shown in Table 6. As mentioned above, these results have been computed using the Atlas.ti software [2]. From the results of Table 6 we observe that the values of α binary are generally high, providing evidences of agreement between the judges-i.e. Coder 1 and Coder 2 as described in Subsection 3.4-in the application of the different semantic domains. For iteration 1, the value of α binary in 9 out of 10 of the semantic domains is above the minimal threshold 0.667. Even more, in 8 out of 10 of the semantic domains, the value is above 0.80, showing statistical evidence of reliability in the evaluations. In iteration 2, the values of α binary increased or remained equal in 8 out of 10 of the semantic domains. In this way, 8 out of 10 of the semantic domains present values of α binary above 0.667 in iteration 2 and, actually, all of them are above 0.8. However, we also observe that there exists two semantic domains that present more difficulties for being detected, namely P02 and P07.
In the case of P02, we observe that its α binary values are consistently below the threshold 0.667 (but increasing from iteration 1 to iteration 2). This evidences that, despite that the agreement of the judges is clearly better than chance, the limits of this domain are not well-defined. Indeed, this domain measures the needs of the analyzed companies of 'higher quality' in several aspects of their development and operational cycle, which is intrinsically a very vague concept that opens the door to several interpretations from the judges. For this reason, in order to clarify the limits and applicability of this domain, future research is needed for proposing quantitative metrics of quality.
Finally, the case of P07 is very special since it dramatically decreases from iteration 1 to iteration 2 (where it is actually negative). The point is that, as shown in Figure 14 in fact, in iteration 2 there is only one evaluation from one of the judges that assigns this semantic domain, in contrast with the 15 evaluations obtained in iteration 1. For this reason, there are not enough data for evaluating this domain in iteration 2 and, thus, this result can be attributed to statistical outliers. With respect to the research question RQ2 (results), the obtained values of α binary for the semantic domains described in Section 4.2 are shown in Table 7. From these results, we observe that in iteration 1 all the semantic domains achieved values of α binary higher than the minimum threshold of significance of 0.667. Furthermore, 9 out of 10 of the domains achieved values higher than 0.8, showing statistical evidence of the reliability of the evaluations. Moreover, in iteration 2 the judged attained perfect agreement of the choice of the semantic domains, giving rise to a value of α binary of 1.0 in all the domains. This evidences that the boundaries of these semantic domains are perfectly stated and that it straightforward to determine whether to apply a code from a semantic domain or not.
Remark 5.1. As mentioned in Subsection 3.4, in our study the quotations were pre-established before the evaluation of Coders 1 and 2. In this way, most of the matter of the interviews is not selected as eligible since it is considered to be irrelevant for the analysis. This is consistent with the fact that relevant information in oral transmission is usually sparser than in written media due to the cognitive needs of the speaker for elaborating an speech on the fly. However, a limitation of the used software Atlas.ti [2] forces to consider the whole corpus in the ICA analysis and does not allow us to remove the irrelevant matter from the analysis. This implies that the shown results are computed on the whole corpus.

5.2.
Results for cu-α. In our case of study, the results of cu-α for the research question RQ1 (problems) are shown in Table 8. From these results, we observe that, in the iteration 1, the value of cu-α for 9 out of 10 of the semantic domains is above the minimal threshold of 0.667. Moreover, 7 out of 10 are above the confident threshold of 0.8. This shows that, regarding the scope of application of the codes within a particular domain, the results obtained in the study are sound and well-posed. Indeed, 6 out of 10 of the coefficients are exactly 1.0, meaning that there existed a perfect matching in the decisions of the judges. Even more, in the iteration 2, all the obtained coefficients consistently improved until reaching the value 1.0, that is, perfect matching within all the domains.  It is worth mentioning that, due to the limitations of the used software, some of the perfect matches were labeled as N/A. The reason is that Atlas.ti considers that the results present not enough variability for computing cu-α. For instance, domain P04 in iteration 1 was never chosen in by any judge. In the same vein, at domains P02, P06 and P07 of iteration 2, only one code within them was used. These claims can be checked in Figure 15. For these reasons, the software displays that the results are not available, despite that, strictly speaking, the cu-α coefficient is 1.0, as stated in Table 8.
On the other hand, the results of cu-α for research question RQ2 (results) are shown in Table 9. Overall, the values of cu-α for this research question are high. Indeed, in iteration 1, the judges obtained perfect matching in the choice of the codes within a semantic domain, implying a value of cu-α of 1.0. In iteration 2, the results follow the same lines, but we find a couple of values that deserve an explanation. These are the values of cu-α for the semantic domains R03 and R04, whose evaluations can be checked in Figure 16. In the case of R04, as mentioned above, it was labelled with N/A by Atlas.ti because the software considered that there was not enough statistical variability. In this particular case, the two judges picked only one code from the semantic domain, but they agreed in the choice, so this is, indeed, a perfect matching.
In the case of R03, the small value of cu-α is produced by the combination of two facts. First, from the three codes within R03, only two are used by the judges (R03a "Process automation: efficiency, optimization, productivity" and R03c "Reduce IT cost"). However, the use of these two codes is deeply unbalanced: the former code received all the evaluations from the judges except a short quotation that was coded by one judge with the later code. This unbalanced situation

5.3.
Results for Cu-α. We show, in Table 10, the values of Cu-α for iterations 1 and 2 and for the research questions RQ1 (problems) and RQ2 (results). With respect to the RQ1 (problems), in iteration 1, we obtained a global Cu-α coefficient of 0.67, which is slightly above the lower threshold of applicability of 0.667. This is compatible with the observation of Section 5.1 that, for this iteration, α binary reached significantly low values for semantic domains P02 and P09. This is an evidence that the boundaries of some semantic domains in the first version of the codebook were not well-defined and their descriptions presented a certain amount of overlapping. Nevertheless, this problem was overcome with the new version of the codebook. In iteration 2, the value of Cu-α increased until 0.905. This is clearly above the threshold of 0.8 reported in the literature for getting statistical significance. This evidences that the review process of the codebook detected the flaws in the description of the limits of applicability of the semantic domains, which led to clearer and bounder definitions.
With respect to the research question RQ2 (results), we observed that Cu-α attained very high values, higher than 0.8, in both iterations. This confirms the trend previously observed that the semantic domains of these research question are clearly stated. Indeed, the value of Cu-α increased in iteration 2 with respect to the value of iteration 1, in consonance with the previous observation for RQ1.

RELATED WORK
Chronologically, although the work by Iden et al. [24] did not explicitly mention DevOps, we can consider that is one of the first papers that empirically analyzed the conflict between development and operations teams when they have to collaborate. This follows the idea initiated by P. Debois [10] and Flickr employees [1]. Iden et al. [24] used the Delphy method (brainstorming with 42 Norwegian IT experts, reduction and ranking) to provide a key baseline for analyzing the problems that reveal the lack of cooperation between developers and IT operations personnel. Some years later, Erich et al. [16] performed a systematic mapping study to analyze the benefits of this cooperation between development and operations, and Smeds et al. [46] interviewed 13 subjects in a software company adopting DevOps to research about the main defining characteristics of DevOps and the perceived impediments to DevOps adoption.
Lwakatare et al. [33] used multi-vocal literature and three interviews from one case company to describe what DevOps is and DevOps practices according to software practitioners. Riungu-Kalliosaari et al. [40] conducted a qualitative multiple-case study and interviewed the representatives of three software development organizations in Finland to answer how industry practitioners perceive the benefits of DevOps practices in their organization and how they perceive the adoption challenges related to DevOps.
In the following years a greater number of studies conducted empirical research by involving more and more companies. Erich et al. [17] performed an exploratory study on six companies to answer again about what DevOps is, the effects when DevOps is being practiced, and how DevOps is being implemented and what supportive factors exist. Kuusinen et al. [30] also conducted a case study in a large Danish software company to identify challenges, impediments, and barriers that a large company faces when transitioning towards DevOps. Senapathi et al. [45] conducted an exploratory case study that explored DevOps implementation in a New Zealand product development organisation to research the main drivers for adopting DevOps, the engineering capabilities and technological enablers of DevOps, and the benefits and challenges of using DevOps. This work, for the first time, researched why companies are instilling a DevOps culture, although only in a large organization in the Finance/Insurance sector. Recently, Leite et al. [31] has published an exhaustive survey on DevOps concepts and challenge, in which through a method inspired on systematic literature review and grounded theory, they analyze practical implications for engineers, managers and researchers.
These academic studies, all of them based on empirical research, provide a key baseline for future studies with a broader scope until achieving the saturation for qualitative studies. They mainly focused on DevOps definition, practices, benefits and challenges (see Table 11), but they did not delve into the motivation, drivers, and problems that lead companies to adopt DevOps, as well as the expectations of this cultural and organizational transformation. Erich et al. [17] pointed out the need of more experimental studies and quantitative studies to verify the state of DevOps. In this regards, the report made by DORA (DevOps Research & Assessment association) [39] and the report made by Puppet and Splunk [49], analyzed data of survey questionnaires over 30,000 technical professional worldwide, respectively. The first one identifies a set of software delivery performance profiles (elite, high, medium and low performance) and relates DevOps practices with these profiles. The second one identifies five stages of DevOps evolution (aka. the DevOps evolutionary model) and establishes the practices that define and/or contribute to success in that stage. These reports also provide a valuable information for companies as they provide a global picture, but they do not respond to why many business are instilling a DevOps culture.
Between quantitative studies with over 30,000 respondents and qualitative studies with 1-6 case studies, we presented this multiple case study of 30 software-intensive companies with interviews to +30 relevant stakeholders that satisfies the criteria for the saturation for qualitative studies. Finally, none of the mentioned related work applies methods for evaluating the reliability and consistency of their study findings based on coding the qualitative data into various themes, i.e. thematic analysis. Generally, in software engineering, when thematic analysis is used in qualitative research, inter-coder reliability measures are not calculated, and thus, authors bias in findings may exist and it remains undetected.

CONCLUSIONS
This paper provides empirical evidence about the real reasons why companies move to DevOps and what results they expect to obtain when adopting DevOps culture. This paper describes empirical research on practicing DevOps through a multiple case study of 30 multinational softwareintensive companies that consists of interviews to relevant stakeholders. To improve the validity of this exploratory case study we applied various strategies, such as data triangulation, member checking, rich description, clarify bias, and report discrepant. It is especially relevant the method used in this paper to reduce authors bias: the inter-coder agreement, based on some variants of Krippendoff's α coefficient. All these strategies mitigate the problems inherent in qualitative research and reinforce our findings.
The main problem that motivates software companies to adopt DevOps is that delivering software takes too much time. DevOps culture and practices promote higher levels of process automation and efficiency and team effectiveness and collaboration. This leads to faster time-to-market and also contributes to improve software quality and customer satisfaction.
We have discovered some patterns and anti-patterns about the reasons why companies are instilling a DevOps culture in their organization (see Figure 17).
Some patterns focus on a cultural and organizational perspective, emphasizing silos between dev & ops and the lack of collaboration and the need of breaking these silos and improve collaboration (Silos remover), whereas other patterns focus on team and process efficiency, emphasizing the lack of processes automation and standardized technology stacks, infrastructure, methodologies, etc. and the need of optimizing and automating processes (Efficiency enabler). What is clear is the most organizations adopt DevOps with the goal of accelerate time-to-market (Time-to-market rusher) while they deliver high quality products (Agility & Quality juggler).
Some anti-patterns are about the fact that some organizations adopt DevOps by a trend or hype (Headless chicken anti-pattern) and the fact that they adopt DevOps simultaneously they are addressing a large change (e.g. reference architecture or infrastructure of core applications) likely incurring in a Snowballing anti-pattern. As further work we plan providing more empirical evidence about other research questions, such as, team topologies, key performance attributes such as lead time, deployment frequency, and mean time to recovery, and other barriers to adopt DevOps, among others.

ACKNOWLEDGEMENT
This study is also being performed thanks to Vass, Clarive, Autentia, Ebury, Carrefour, Vilt, IBM, AtSistemas, Entelgy, Analyticalways, Mango eBusiness, Adidas, Seur, Zooplus, as well as other participating companies. APPENDIX A. APPENDIX. THEORETICAL FRAMEWORK FOR α COEFFICIENTS In this Appendix, we introduce a novel interpretation that unifies the different variants of the α coefficient in a common framework. These coefficients that may be found in the literature are presented as unrelated and a kind of ad hoc formulation for each problem is provided.
The key point of this section is that we will show that these versions can be translated to simpler and universal version of Krippendoff's α. For this purpose, we will formulate the α coefficients in terms of some 'meta-codes', that we will call 'labels'. In each situation, we will provide a well-defined algorithm that translates from semantic domains and codes (the units of judgment considered in thematic analysis) into labels. In this way, after this translation, all the coefficients reduce to the same mathematical computation of the universal α coefficient for labels.
In order to lighten this section, the mathematical formulation of this universal α coefficient has been moved to Appendix B. Nevertheless, for convenience, let us recall some notations introduced there that will be used along this section. We fix a finite set Λ of labels and we are dealing with a collection J 1 , . . . , J n of judges that will evaluate a set of items I 1 , . . . , I m . Each of the judges, J α , evaluates the item I β with a subset ω α,β ⊆ Λ. The result of the evaluation process is gathered in a set Ω = ω α,β for 1 ≤ α ≤ n and 1 ≤ β ≤ m.
From Ω, we can compute Krippendoff's coefficient, α = α(Ω), which is a real number with 0 ≤ α ≤ 1. Such a quantity is interpreted as a measure of the degree of agreement that is achieved out of the chance. The bigger the α is, the better agreement is observed. A common rule-of-thumb in the literature [28] is that α ≥ 0.667 is the minimal threshold required for drawing conclusions from the data. For α ≥ 0.80, we can consider that there exists statistical evidence of reliability in the evaluations.
However, this ideal setting, as described in Appendix B, might be too restrictive for the purposes of content analysis (particularly, as applied by the Atlas.ti Software [2]). The most general setting of content analysis is as follows. We have a collection of s > 1 semantic domains, S 1 , . . . , S s . A semantic domain defines a space of distinct concepts that share common meanings (for a concrete example, check our semantic domains in Subsection 4.1.1). Subsequently, each semantic domain embraces mutually exclusive concepts indicated by a code. Hence, for 1 ≤ i ≤ s, the domain S i for 1 ≤ i ≤ s, decomposes into r i ≥ 1 codes, that we will denote by C i 1 , . . . ,C i r i . As pointed out in the literature, for design consistency these semantic domains must be logically or conceptually independent. This principle translates into the fact that there exists no shared codes between different semantic domains. Now, the data under analysis (e.g. scientific literature, newspapers, videos, interviews) is divided into items, which in this context are known as quotations, that represent meaningful parts of the data by their own. The decomposition may be decided by each of the judges (so different judges may have different quotations) or it may be pre-established (for instance, by the codebook creator or the designer of the ICA study). In the later case, all the judges share the same quotations so they cannot modify the limits of the quotations and they should evaluate each quotation as a block. In order to enlighten the notation, we will suppose that we are dealing with this case of pre-established quotations. This is actually the setting of our study (see Subsection 3.4). Indeed, from a mathematical point of view, the former case can be reduced to this version by refining the data division of each judge to get a common decomposition into the same pieces.
Therefore, we will suppose that the data is previously decomposed into m ≥ 1 items or quotations, I 1 , . . . , I m . Observe that the union of all the quotations must be the whole matter so, in particular, irrelevant matter is also included as quotations. Now, each of the judges J α , 1 ≤ α ≤ n, evaluates the quotation I i , 1 ≤ i ≤ m, assigning to I i any number of semantic domains and, for each chosen semantic domain, one and only one code. No semantic domain may be assigned in the case that the judge considers that I β is irrelevant matter, and several domains can be applied to I β by the same judge.
Hence, as byproduct of the evaluation process, we obtain a collection of sets Σ = σ α,β , for 1 ≤ α ≤ n and 1 ≤ β ≤ m. Here, σ α,β = C i 1 j 1 , . . . ,C i p j p is the collection of codes that the judge J α assigned to the quotation I β . The exclusion principle of the codes within the semantic domain means that the collection of chosen semantic domains i 1 , . . . , i p contains no repetitions.
Remark A.1. To be precise, as proposed in [29], when dealing with a continuum of matter each of the quotations must be weighted by its length in the observed and expected coincidences matrices (see Appendix B). This length is defined as the amount of atomic units the quotation has (say characters in a text or seconds in a video). In this way, (dis)agreements in long quotations are more significant than (dis)agreements in short quotations. This can be easily incorporated to our setting just by refining the data decomposition to the level of units. In this way, we create new quotations having the length of an atomic unit. Each new atomic quotation is judged with the same evaluations as the old bigger quotation. In the coefficients introduced below, this idea has the mathematical effect that, in the sums of Equation (1, Appendix B), each old quotation appears as many times as atomic units it contains, which is the length of such quotation. Therefore, in this manner, the version explained here computes the same coefficient as in [29].
In order to quantify the degree of agreement achieved by the judges in the evaluations Σ, several variants of Krippendorff's α are proposed in the literature [29,28]. For the purposes of this study, we will apply the variants described below.
A.0.1. The coefficient α binary . The first variation of the Krippendorff's α coefficient is the socalled α binary coefficient. This is a coefficient that must be computed on an specific semantic domain. Hence, let us fix a semantic domain S i for some fixed i with 1 ≤ i ≤ s. The set of considered items to be judged is exactly the set of (prescribed) quotations I 1 , . . . , I m . However, the set of labels will have only two labels, that semantically represent 'voted S i ' and 'did not vote S i '. Hence, we take For the assignment of labels to items, the rule is as follows. For 1 ≤ α ≤ n and 1 ≤ β ≤ m, we set ω α,β = {1} if the judge J α assigned some code of S i to the quotation I β (i.e. if C i j ∈ σ α,β for some 1 ≤ j ≤ r i ) and ω α,β = {0} otherwise. Observe that, in particular, ω α,β = {0} if J α considered that I β was irrelevant matter. From this set of evaluations, Ω S i binary = ω α,β , α binary is given as In this way, the coefficient α S i binary can be seen as a measure of the degree of agreement that the judges achieved when choosing to apply the semantic domain S i or not. A high value of α S i binary is interpreted as an evidence that the domain S i is clearly stated, its boundaries are well-defined and, thus, the decision of applying it or not is near to be deterministic. However, observe that it does not measure the degree of agreement in the application of the different codes within the domain S i . Hence, it may occur that the boundaries of the domain S i are clearly defined but the inner codes are not well chosen. This is not a task of the α S i binary coefficient, but of the cu-α S i coefficient explained below.
Remark A.2. Empirically, we discovered that the semantic that the software Atlas.ti [2] applies for computing α binary (and for the coefficient cu-α introduced in Section A.0.2) is the one explained in this section. However, to our understanding, this behaviour is not consistent with the description provided in the corresponding user's guide.
A.0.2. The coefficient cu-α. Another variation of the Krippendorff's α coefficient is the so-called cu-α coefficient. As the previous variation, this is a coefficient that is computed for each semantic domain, say S i for some 1 ≤ i ≤ s. Suppose that this semantic domain contains codes C i 1 , . . . ,C i r . As always, the set of considered items is the set of quotations. However, the collection of labels is now a set Λ = {C 1 , . . . , C r } .
Semantically, they are labels that represent the codes of the chosen domain S i .
For the assignment of labels to items, the rule is as follows. For 1 ≤ α ≤ n and 1 ≤ β ≤ m, we set ω α,β = C k if the judge J α assigned the code C i k of S i to the item (quotation) I β . Recall that, from the exclusion principle for codes within a semantic domain, the judge J α applied at most one code from S i to I β . If the judge J α did not apply any code of S i to I β , we set ω α,β = / 0. From this set of judgements Ω S i cu = ω α,β , cu-α is given as cu-α S i = α(Ω S i cu ).
Remark A.3. As explained in Remark B.1 of Appendix B, for the computation of the observed and expected coincidence matrices, only items that received at least to evaluations with codes of S i from two different judges count. In particular, if a quotation is not evaluated by any judge (irrelevant matter), received evaluations for other domains but not for S i (matter that does not corresponds to the chosen domain) or only one judge assigned to it a code from S i (singled-voted), the quotation plays no role in cu-α. This limitation might seem a bit cumbersome, but it could be explained by arguing that the presence/absence of S i is measured by α S i binary so it will be redundant to take it into account for cu-α S i too.
A.0.3. The coefficient Cu-α. The last variation of Krippendorff's α coefficient that we consider in this study is the so-called Cu-α coefficient. In contrast with the previous coefficients, this is a global measure of the goodness of the partition into semantic domains. Suppose that our codebook determines semantic domains S 1 , . . . , S s . As always, the set of considered items is the set of quotations, but the collection of labels is the set Λ = {S 1 , . . . , S s } .
Semantically, they are labels representing the semantic codes of our codebook.
We assign labels to items as follows. Let 1 ≤ α ≤ n and 1 ≤ β ≤ m. Then, if σ α,β = C i 1 j 1 , . . . ,C i p j p , we set ω α,β = S i 1 , . . . , S i p . In other words, we label I β with the labels corresponding the semantic domains chosen by judge J α for this item, independently of the particular code. Observe that this is the first case in which the final evaluation Ω might be multivaluated. From this set of judgements, Ω Cu = ω α,β , Cu-α is given as Cu-α = α(Ω Cu ).
In this way, Cu-α measures the degree of reliability in the decision of applying the different semantic domains, independently of the particular chosen code. Therefore, it is a global measure that quantifies the logical independence of the semantic domains and the ability of the judges of looking at the big picture of the matter, only from the point of view of semantic domains. APPENDIX B. APPENDIX. UNIVERSAL KRIPPENDORFF'S α COEFFICIENT In this appendix, we rephrase Krippendorff's α for a wide class of judgements. This gives rise to a universal Krippendorff's α coefficient formulated for assignments of 'meta-codes' called 'labels'. This formulation is very useful for the unified formulation of several variants of α, as introduced in Appendix A. For an historical description of this coefficient, check [28].
In the context of Inter-Coder Agreement analysis, we are dealing with n > 1 different judges (also known as coders), denoted by J 1 , . . . , J n , as well as with a collection of m ≥ 1 items to be judged (also known as quotations), denoted I 1 , . . . , I m . We fix a set of N ≥ 1 admissible 'metacodes', called labels, say Λ = {l 1 , . . . , l N }. The task of each of the judges J α is to assign, to each item I β , a collection (maybe empty) of labels from Λ. Hence, as byproduct of the evaluation process, we get a set Ω = ω α,β , for 1 ≤ α ≤ n and 1 ≤ β ≤ m, where ω α,β ⊆ Λ is the set of labels that the judge J α assigned to the item I β . Recall that ω α,β is not a multiset, so every label appears in ω α,β at most once.
In other words, o i, j counts the number of (ordered) pairs of responses of the form (ω α,β , ω α ,β ) ∈ Ω × Ω that two different judges J α and J α gave to the same item I β and such that J α included l i in his response and J α included l j in his response, or viceversa.
Remark B.1. Suppose that there exists an item I β that was judged by a single judge, say J α . The other judges, J α for α = α, did not vote it (or, in other words, they voted it as empty), so ω α ,β = / 0. Then, this item I β makes no contribution to the calculation of o i, j since there is no other judgement to which ω α,β can be paired. Hence, from the point of view of Krippendoff's α, I β is not taken into account. This causes some strange behaviours in the coefficients below that may seem counterintuitive.
From these counts, we construct the matrix of observed coincidences as M o = o i, j N i, j=1 . By its very construction, M o is a symmetric matrix. From this matrix, we set t k = ∑ N j=1 o k, j , which is (twice) the total number of times that the label l k ∈ Λ was assigned by any judged. Observe that t = ∑ N k=1 t k is the total number of judgments. In the case that each judge evaluates each item with a single non-empty label, we have t = nm.
On the other hand, we can construct the matrix of expected coincidences, M e = e i, j N i, j=1 , where The value of e i, j might be though as the average number of times that we expect to find a pair (l i , l j ), when the frequency of the label l i is estimated from the sample as t i /t. Again, M e is a symmetric matrix.
Finally, let us fix a pseudo-metric δ : Λ × Λ → [0, ∞) ⊆ R, i.e. a symmetric function satisfying the triangle inequality and with δ (l i , l i ) = 0 for any l i ∈ Λ (recall that this is only a pseudo-metric since different labels at distance zero are allowed). This metric is given by the semantic of the analyzed problem and, thus, it is part of the data used for quantifying the agreement. The value δ (l i , l j ) should be seen as a measure of how similar the labels l i and l j are. A common choice is socalled discrete metric, given by δ (l i , l j ) = 0 if i = j and δ (l i , l j ) = 1 otherwise. The discrete metric means that all the labels are equally separated. This is the underlying assumption that we will apply in our study. However, subtler metrics may be used for extracting more semantic information from the data (see [29]).
From these computations, we define the observed disagreement, D o , and the expected disagreement, D e , as This quantities measure the degree of disagreement that is observed from Ω and the degree of disagreement that might be expected by judging randomly, respectively.
Remark B.2. In the case of taking δ as the discrete metric, we have another interpretation of the disagreement. Observe that, in this case, since δ (l i , l i ) = 0 we can write the disagreements as The quantity A o = ∑ N i=1 o i,i (resp. A e = ∑ N i=1 e i,i ) can be understood as the observed (resp. expected) agreement between the judges. In the same vein, t = ∑ N i, j=1 o i, j = ∑ N i, j=1 e i, j may be seen as the maximum achievable agreement. Hence, in this context, the disagreement D o (resp. D e ) is indeed the difference between the maximum possible agreement and the observed (resp. expected) agreement.
From these data, Krippendroff's α coefficient is defined as From this formula, observe we following limitting values: • α = 1 is equivalent to D o = 0 or, in other words, it means that there exists perfect agreement in the judgements among the judges.
• α = 0 is equivalent to D o = D e , which means that the agreement observed between the judgements is entirely due to chance.
In this way, Krippendorff's α can be interpreted as a measure of the degree of agreement that is achieved out of the chance. The bigger the α is, the better agreement is observed.
Remark B.3. Observe that α < 0 may only be achieved if D o > D e , which means that there is even more disagreement than the one that could be expected by chance. This implies that the judges are, consistently, issuing different judgements for the same items. Thus, it evidences that there exists an agreement between the judges to not agree, that is, to fake the evaluations. On the other hand, as long as the metric δ is non-negative, D o ≥ 0 and, thus, α ≤ 1.