{'fraction_non_alphanumeric': 0.02678887394720079, 'fraction_numerical': 0.007803100007077642, 'mean_word_length': 4.983800952885124, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ease on Down the Code: Complex Collaborative Qualitative Coding Simplified with 'Code Wizard'

This paper describes the design and development of a preliminary qualitative coding tool as well as a method to improve the process of achieving inter-coder reliability (ICR) in small teams. Software applications that support qualitative coding do not sufficiently assist collaboration among coders and overlook some fundamental issues related to ICR. We propose a new dimension of collaborative coding called"coders' certainty"and demonstrate its ability to illustrate valuable code disagreements that are missing from existing approaches. Through a case study, we describe the utility of our tool, Code Wizard, and how it helped a group of researchers effectively collaborate to code naturalistic observation data. We report the valuable lessons we learned from the development of our tool and method: (1) identifying coders' certainty constitutes an important part of determining the quality of data analysis and facilitates identifying overlapping and ambiguous codes, (2) making the details of coding process visible helps streamline the coding process and leads to a sense of ownership of the research results, and (3) there is valuable information hidden in coding disagreements that can be leveraged for improving the process of data analysis.


INTRODUCTION
Collaborative qualitative coding is an iterative and often lengthy process mainly consisting of reading, interpreting, classifying, and analyzing qualitative data such as interview transcripts and field notes. The length of the coding process is correlated primarily with the volume of such data and the number of individual collaborating coders. There are various approaches to conduct collaborative coding inspired by different philosophical paradigms such as positivism and interpretivism.
In addition, we introduce a new dimension in collaborative coding, "coders' certainty," and discuss how it improved the collaborative process of achieving coders' reliability in a small team. We describe our method of leveraging correlated disagreements to support achieving higher agreement threshold (i.e., higher ICR). We also demonstrate Code Wizard's utility by discussing how it helped a group of researchers collaborate on coding observation field notes. It is important to point out that qualitative coding includes various components, but in this paper, we focus exclusively on the issues related to collaborative coding from positivist perspective, and not other aspects such as segmentation/unitization, and developing categories and themes.
The following research questions guided our research: RQ1: What aspects of collaborative qualitative coding can facilitate the challenges of collaboration? RQ2: How can we use these aspects to design a tool to facilitate the collaborative coding process? We first review the literature of qualitative data analysis with special attention to collaborative coding. We briefly review a few popular qualitative coding software applications. Then, we describe our methodology and the development of Code Wizard and demonstrate its application through a case study. We conclude by discussing our findings, recognizing areas for improvements, and identifying directions for future research.

Collaborative Coding
Collaborative coding is key to providing sound interpretation of qualitative data and is vital to creating valid results. Burla et al. (2008) indicate that for large volumes of data, it is best to have more than one person code for efficiency as it helps increase comprehension, support inter-subjectivity, and provide sound interpretation of the data [3].
Collaborative coding with multiple coders benefits from a diversity of knowledge and skills that can result in more clarity in codes, the main cause of disagreement. The disagreements are often addressed through clarification discussions that may lead to refining coding frame [12 & 14].
Collaborative coding is a response to the challenges of subjectivity of qualitative data analysis. Barbour (2001) indicates that "the greatest potential of collaborative coding lies in its capacity to furnish alternative interpretations and thereby to act as the 'devil's advocate' implied in many of the checklists

Research Project
In June 2016, the largest regional disaster response exercise conducted in the Pacific Northwest took place. Cascadia Rising 2016 was a "four-day, large scale exercise to test response and recovery capabilities in the wake of a 9.0 magnitude CSZ earthquake and tsunami." The exercise involved local, state, tribal, federal partners, nongovernmental organizations, the private sector, and military commands. Over 20,000 emergency managers participated in the exercise in Washington, Idaho, and Oregon [25][26][27].
Our team saw this exercise as an opportunity to study a complex, dynamic, and non-routine event and investigate the obstacles of information sharing and coordination during a disaster event. Our team conducted fieldwork observations at Emergency Operations and Coordination Centers, and the Federal Emergency Management Agency and major military centers for 4 days and collected 8 to 10 hours of field notes per day. Field observations were conducted at state and county locations.
We used Slack [28] to capture observations because it (a) time-stamps the notes, (b) facilitates communication between the observers, (c) has a built-in search feature, and (d) allows researchers at multiple sites to follow and track data collection in real-time and help clarify data when necessary.
The coding team consisted of 5 graduate and undergraduate students from the Departments of Civil Engineering, Cognitive Psychology, Electrical Engineering, and Human-Centered Design & Engineering. An experienced qualitative researcher led the coding process.
Our coding team faced 4 main challenges: (1) the coders were not familiar with disaster recovery; (2) team members changed quarterly, and onboarding new coders every 10-12 weeks was not realistic. Tools such as Dedoose and NVivo require training [19]; (3) the coders' diverse backgrounds and varying experiences in qualitative coding necessitated extra work to reach a common language; and (4) The team found no software that they all were proficient at or could be mastered quickly.
Sidebar 2 in alerting researchers to all potentially competing explanations." She emphasizes the value of collaborative coding because it leads researchers to confront different interpretations of data or codebook in research team meetings, which improves the quality of the qualitative content analysis [14]. Krippendorff (2004Krippendorff ( , 2011) defines reliability as "the degree to which members of a designated community agree on the readings, interpretations, responses to, or uses of given texts or data." While reliability has different components, reproducibility (interchangeably used as inter-coder reliability or interrater reliability) is the minimum requirement to establish reliability in qualitative data analysis. Reproducibility is defined as "the degree to which a process can be replicated by different analysts working under varying conditions, at different locations, or using different but functionally equivalent measuring instruments" [1 and 7]. According to Lombard et al. (2002), ICR is widely used to identify "the extent to which independent coders evaluate a characteristic of a message or artifact and reach the same conclusion" [11]. Barbour (2001) highlights the importance of taking a qualitative approach to identify reasons behind coding disagreements and not solely rely on quantitatively measuring ICR. The authors support the "merit of coding discussions" to improve qualitative analysis [14].

Inter-coder Reliability
While many people use the terms reliability and agreement interchangeably, Krippendorff (2004) notes, "agreement is what we measure; reliability is what we wish to infer from it. In content analysis, reproducibility is arguably the most important interpretation of reliability" [7]. Lombard et al. (2002) also state "[i]t is widely acknowledged that intercoder reliability is a critical component of content analysis and (although it does not ensure validity) when it is not established, the data and interpretations of the data can never be considered valid." ICR can be inferred by measuring agreement level among coders in collaborative analysis [11].
Artstein and Poesio (2008) compared different methods of measuring agreement among coders in collaborative coding such as direct percentage of agreement, Scott's pi, Cohen's kappa, Krippendorff's alpha, and Fleiss' kappa. They do not recommend direct percentage of agreement because it does not consider the chance agreement in its formula. The direct percentage method is also limited to two coders. Except for Krippendorff's alpha and Fleiss' kappa, other methods fail in measuring agreement for more than two coders. Fleiss' kappa is a generalized form of Scott's pi for multiple coders and also much simpler than Krippendorff's alpha approach [15].
Although there is no standard threshold for determining acceptable level of reliability [11], Fleiss' kappa of higher than 0.8 is considered an "almost perfect" level of agreement among coders [11,16].

Challenges of Achieving High ICR
Achieving an acceptable ICR is impacted by various factors. According to Hruschka et al. (2004), a large sample size may increase the complexity of coding and decrease ICR. In addition, "variation in the content, length of response, or number of codes per question may affect the speed at which a team achieves an acceptable level of inter-coder reliability." Finally, "variation in the clarity of the codebook and individual code definitions" are other factors that can influence the reliability process [9].
The use of multiple coders may introduce new challenges. According to Berends and Johnston (2005), in a team of multiple coders, different skill levels, resources, and time demands can slow down the collaborative coding process. However, there is a tradeoff to consider, because inclusion of multiple perspectives in researcher backgrounds plays an important role in discussing the disagreements and refining the coding system. Similarly, Barbour (2001) states that engaging multiple coders improves the clarity of the codebook and helps identify reasons behind coding disagreements [12 and 14]. Burla et al. (2008) categorize the sources for low ICR into two groups: (1) factors related to codes such as weaknesses of coding scheme, definitions, examples, and classification rules, (when codes were mutually exclusive), and (2) factors related to coders' skill levels such as training issues that lead to insufficient understanding or inappropriate application of the coding scheme by one coder [3].
Collaborating remotely with the research team is another reason for low ICR. MacPhail et al. (2016) report that their team had to conduct 10 rounds of collaborative coding over the course of one year to achieve high Cohen's kappa [8]. They state that their main challenge was coders' different understanding of how to use and apply the codes because the coders were located in different countries. Coders needed to use phone and emails to communicate with the rest of the team, which lengthened the coding process. MacPhail et al. also state that "there are limited practical resources available for researchers engaging in a group coding process and interested in ensuring adequate Inter-coder Reliability (ICR); the amount Our research team was required to complete an online disaster recovery and management course offered by the government when they joined the project, which helped with the first challenge.
However, quarterly changes to team composition made the coding process challenging because new team members needed time to understand the codes and each other's perspectives, which was timeconsuming.
When recruiting new project team members each quarter, we did not want to recruit coders based on their knowledge of particular tools. We also did not want to compromise the data collection, data usage, and data analysis by tailoring the study needs around the capabilities of particular software tools.
As Woods et al. [19] indicate, "software can dominate the researchers' understanding of their practices" especially if the qualitative data analysis software tools are used "without a critical and reflexive awareness" of their influences on qualitative research practices.

Software Tools and Visualization Techniques for Qualitative Coding
There are various software applications to support collaborative qualitative coding. Peter Nielsen (2012) evaluated and compared the features of four prominent tools including, NVivo, Atlas.ti, Dedoose, and HyperResearch. Nielsen demonstrated that NVivo, Atlas.ti, and HyperResearch are more useful for individual coding but "are insufficient to support collaborative coding." On the other hand, Dedoose is "fully and transparently collaborative," but it lacks certain features to support data analysis and "does not easily relate codes to each other in a structure (hierarchical or network)" [13].
Woods et al. [19] investigated 763 empirical articles to explore how researchers use Atlas.ti and NVivo and found that although these tools can support multiple phases of the research process, the majority of researchers use them for data management and analysis, "with fewer using it for data collection/creation or to visually display their methods and findings." Some researchers use visuals and colors to help the coding process. For example, Blascheck et al. use a visual analytics approach to integrate various data formats, such as transcripts and videos [22]. The authors create a tabular representation of the data using colors, panels, and Sidebars to provide an overview of all user activities. Quirkos [23] uses colors to categorize and cluster codes and are more focused on individual coding. Aeonium [24] is a coding interface that uses machine learning and visual analytic techniques to facilitate identifying ambiguous data. Aeonium helps direct coders' attention to the ambiguous data that warrants analysis and is more focused on the collaboration aspect of coding.

METHODOLOGY
In this section, we describe the iterative process of developing Code Wizard and illustrate its application to our research project, which is described in the Sidebars. The Sidebars narrate a case study in which we realized the need to improve our collaborative coding methodology and develop a new tool to address our challenges. Sidebar 1 describes the context of our research project. Sidebar 2 gives insight into our data The team made initial attempts to use Dedoose [21] based on the recommendation of our lead researcher. However, we abandoned Dedoose after a few weeks because (1) the team members felt Dedoose was difficult and time-consuming to learn, (2) it was slow to work with, especially for the code disagreement discussion meetings, and (3) it did not offer the data analysis functionalities the team required, such as calculating Fleiss' kappa or other ICR measures for more than two coders.
The team decided not to switch to a new tool such as NVivo or Atlas.ti, because in addition to our unsatisfactory experience with Dedoose, we did not have time to obtain and learn a new tool.
We realized Microsoft Excel was well known to all members of the team and, as a baseline tool, could be easily adopted. However, Excel is not a coding tool and initially the team mostly used it to organize the codes. Experiencing the challenges of an unaided coding process within our small team, we felt the need to add more features to Excel to facilitate our collaboration process ever so slightly. collection procedures, coding team characteristics, and the main challenges that our coding team faced. Sidebar 3 discusses the challenges caused by the structure of our research team.

Research Method
The design and development of Code Wizard had humble beginnings. As we describe in Sidebar 4, our team decided to use Excel to code the textual field notes. We soon realized our coding efforts could benefit from incorporating a few additional features to our Excel sheets, for example to help sort the coded data for team discussions or to produce graphs. Initially, one team member led the design and development of a module by adding features to Excel spreadsheets to address the immediate needs of the coders. Through iterative cycles of use, coders' feedback, and redesign, the tool gradually evolved into a piece of software (See Code 1).
While this design and development process may not seem systematic, our approach has some of the characteristics of practice-based research [29]. We were looking for a way to address our constantly changing environment, had identified some problems (discussed in Section 3.3), saw the need for a tool to support collaborative coding, and started to build such a tool. We deployed the tool in our small coding team and observed how it impacted the team and each individual coder. We then iteratively refined and tested the tool based on the feedback provided by team members or by testing it live in our team discussions.
During the design and development of Code Wizard, we collected three types of data: (1) Data about the tool: we have every version of Code Wizard. This helped us track the changes, show how we modified the tool over time, and understand the reasons that warranted a software update.
(2) Data on the team's coding activities: we collected all versions of the coding activities. These data were important because: (a) the coded data over time provided a way to trace how the coding evolved and how the accuracy or inaccuracy of codes changed as a result of the relationship of the tool with the team, and (b) the applied codes explicitly contributed to the actual disaster recovery research.
(3) Data about the team: team members provided verbal feedback and the developer was present in all of the coding and debriefing discussions as the tool was evolving. To triangulate the verbal feedback, we have access to the emails sent by the coding team to the developer advocating for the changes and modifications of the tool. In addition, some of the team members provided written feedback at the end of the project. For example, one of the coders wrote, "[...] the [Excel spread]sheets were better in helping us compare coder's codes temporally within one screen." Another coder wrote, "The sheets worked well with less overhead than Dedoose, which was the platform we originally tried using for the project." Data Analysis: The primary analysis was to take verbal and written feedback from the team and interpret it such that the tool could resolve the raised issues by the team members.

What is Code Wizard?
As described in Sidebar 2, our team initially experienced four main challenges: (1) the domain of our study was disaster recovery and management, and not all of the coders were familiar with that domain; (2) the coding team consisted of 5 graduate and undergraduate students from various backgrounds. However, the team members changed quarterly, and onboarding new coders every 10-12 weeks was not realistic. This was important because tools such as Dedoose and NVivo are sophisticated and require training [19]; (3)  common language, and (4) the team found no software that they all were proficient at or could be mastered quickly.
In addition to these 'team and software-related' challenges, our team identified three 'process-related' challenges, described in Section 3.3. We realized that our team required some software features that were either missing from the existing tools for collaborative coding or were too complex to learn for our project. Our team's fallback plan was to use Excel to organize the data and codes, and from there, we gradually programmed Excel based on the unfolding needs of the team.
Code Wizard is a visualization tool embedded in Excel. 1 It initially consisted of two Excel files: a pre-formatted spreadsheet for individual coders to use independently, and another spreadsheet that aggregates the individual coding spreadsheets and automatically sorts and compares coded data from individual coders, calculates ICR, and produces visualizations. The aggregated spreadsheet is primarily for the coding lead to use after individual coders finish coding. Below, we discuss each of these files in details. The individual coding file includes three spreadsheets. The first is the main spreadsheet tab where the units of analysis are loaded and sorted in rows and the codes (categories) are organized in a dropdown list. Code Wizard designates one column for primary codes and one column for secondary codes. We will discuss the primary and secondary codes in Section 3.3. Coders can click on the dropdown lists in front of each unit of analysis to select the codes. There are hyperlinks in front of each unit of analysis that take the coders to the source of textual data if they need it. In addition, a "Help" hyperlink directs the coders to the codebook instructions when needed. The second spreadsheet tab includes the codebook and instructions.
Code Wizard assigns a unique color to each code. Finally, the third spreadsheet tab includes the textual data as a reference (Fig. 1).  The aggregated spreadsheet file is primarily used by the coding lead. After the individual coders finish coding, they submit it to the lead. Then, the lead collects all the individual spreadsheets and copies the codes into the aggregated sheet. Code Wizard will then automatically organize and sort the codes, calculate the ICR, and generate the visualizations. Similar to the individual spreadsheets, the aggregated sheet includes a tab for the codebook, a tab for each individual coder's coded data, a tab for the source of data, and two tabs for visualizations. These features and tabs help the lead and the team during discussion sessions.

Collaborative Coding Challenges and Solutions
In this section, we describe each challenge we encountered during our collaborative coding process, followed by our solution. Fig. 1 shows our codebook, which includes seven mutually exclusive codes. We will discuss our results in Section 4.

First Challenge: Discovering Ambiguous
Codes. The initial discussions among coders and the results of sample coding revealed that coders were not fully certain about the codes they had assigned to each unit of analysis. Some of the codes had created more confusion than others. However, the sheer act of assigning one code to a unit of analysis conceals the uncertainties behind that code. We believe there is a difference between assigning a code with high confidence and assigning a code because no other code fits.
As noted previously in Section 2, current methods of measuring ICR reduce the root causes of disagreements to a simple measurement of ICR across all coders. Thus, we were determined to further explore the reasons for coding discrepancies and disagreements and find a way to discern the ambiguous codes and evaluate the degree to which those codes had caused confusion.
Addressing the First Challenge: Coders' Certainty, A Dimension to Support Collaborative Coding. In the first round of coding, we asked individual coders to code each unit of analysis twice. We thought this double coding would help evaluate the uncertainty of individual coders as well as the entire team. The first code should be what the coders deemed as the best fit to the unit of analysis, which we call the primary code. A secondary code would show whether some different codes could potentially be assigned to that same unit of analysis. If coders were absolutely certain about their primary code, then we asked them to select the same code for the secondary code. Otherwise, coders assigned a different secondary code that would be a fit.
We prototyped this coding technique in a spreadsheet ( Fig. 2) with the following main features: (1) we designated two separate columns for primary and secondary codes next to each other, and loaded all the codes to these columns and added a dropdown list to simplify the code selection process; (2) each code was colored differently to expedite locating the different primary and secondary codes; (3) for each unit of analysis, there was a direct link to the source data for coders' reference; (4) a "Help" button was included that linked to the codebook including the description of the codes, examples, instructions, and coding rules.
These features streamlined and accelerated the double coding process by allowing each coder to see all excerpts, access category definitions with one click, and select the codes from the drop down, pre-populated lists.  When individual coders finished coding, our prototype aggregated all the codes and displayed them in one place, so the team could easily see the disagreements using visual cues (Fig. 3). For example, as can be seen in the first row in Fig. 3, coders have very high certainty about Source Breakdown (blue cells), with only one coder selecting a different secondary code for it. However, most coders were not certain about Content Breakdown (yellow cells) and selected a different secondary code for it.
We programmed this aggregated spreadsheet to calculate ICR using Fleiss' kappa statistical index. We preferred Fleiss' kappa method because (1) it is suitable when the number of coders exceeds two, (2) it is simpler and easier to understand compared to alternative methods such as Krippendorff's alpha, (3) it is a stronger method because, unlike many other methods that use direct percentage agreement, it considers the possibility of random agreement in its formula. An acceptable kappa should be equal or greater than 0.8 under Fleiss' method [15,16]. We only used primary codes to calculate ICR because Fleiss' kappa (and most other methods) is based on one code. Our ICR at the first round of coding was 0.52, which is below the acceptable kappa of 0.8. Thus, it was clear our team needed to resolve the disagreements.
After aggregating all the codes and calculating ICR, Code Wizard measured certainties by generating a graph illustrating the coders' certainty per code (Fig. 4). Code Wizard searched for all the individual units whose primary codes were similar, and then compared the primary with the secondary codes for each unit. For example, Code Wizard looked at all the units with Coordination Breakdown as the primary code and then checked whether the primary and secondary codes were similar. As can be seen in Fig. 4, the team had low certainty on Content Breakdown (56 percent), meaning that only 56 percent of the times where Content Breakdown was assigned as the primary code did the coders also assign it as the secondary code. On the other hand, the team was absolutely certain on using Coordination Breakdown, which means 100 percent of the times this code was selected as the primary code, it was also selected as the secondary code. Fig. 4 was particularly useful because it highlighted the codes that needed further discussion, and it guided the team's discussions on the low certainty codes. In addition, our prototype automatically generated the certainty graph for each individual coder (Fig. 5). For example, coder 2 was highly certain about Coordination Breakdown (Green bar) and Paper Mechanism Issue (Orange bar), but had low certainty on Content Breakdown and Disparate Systems. As Fig. 5 illustrates, coder 2 was absolutely certain about Paper Mechanism Issue (Orange bar), which is much higher than the team's certainty rate of 67%, as shown in Fig. 4. This information was helpful in discovering ambiguous codes. This graph helped the team in two ways: (1) individual coders could see where they struggled the most and determine how they needesd help with specific codes, and (2) individual coders could quickly see how their uncertainties compared to others and the team as a whole, and decide where to focus efforts to reconcile discrepancies.
Figs. 1 through 5 helped our team quickly and efficiently discover their uncertainties, disagreements, and the ambiguous codes. However, we needed more details to support the team and collaboration process more effectively.
3.3.2 Second Challenge: Discovering the Overlapping Codes. Although Figs. 1 through 5 revealed code ambiguities and showed overlap between some codes, they do not show which codes overlap and the degree to which these codes overlap. These details are important because in collaborative coding we aim for creating mutually exclusive codes to achieve higher agreement.
This problem is explained with an example in Fig. 6. For Unit 1, primary code A is assigned four times, and similarly, code B is assigned four times for Unit 2. From the perspective of existing coding approaches, Units 1 and 2 have the same level of agreement because four out of five coders assigned the same primary codes to these units of analysis.
However, there is more valuable information in Fig. 6. For Unit 1, the majority of coders assigned code C as the other possible code (Unit 1), but for Unit 2 secondary codes varied, such that codes A, B, C, and D were all used. For Unit 1, we can infer that there might be a strong overlap between codes A and C, especially if this pattern repeats in the coding results. However, code B might be too broad or defined poorly. We wanted to further explore these patterns and see how they could help our team reconcile the disagreements.
In existing collaborative coding methods, coders meet at this point to discuss the disagreements and attempt to find the ambiguous codes. They go over the coded data line by line to find the disagreements and discuss the rationale for their code assignments. Then, they decide whether to revise the code definitions, or take other actions such as adding, removing, or merging the codes. As noted earlier, without Code Wizard this step can take several months and many iterations of collaborative coding especially for large datasets with multiple coders [8, 9, and   Addressing the Second Challenge: Correlated Uncertainty. Our goal was to use the valuable details from double coding to help the team quickly and efficiently find the overlapping codes. Thus, we programmed our spreadsheet prototype such that it (1) compared the primary and secondary codes across all coders to identify rows where secondary codes were different than primary codes or rows, (2) determined the correlation between the primary and secondary codes and, (3) displayed the correlations in a color-coded matrix (Fig. 7).
For example, for all the units for which Disparate Systems (purple) was chosen as the primary code (fourth row), the following codes were selected as secondary: Computational Mechanism Issues (pink) 8% of times, Coordination Breakdown (green) 8% of times, and Disparate Systems 85% of times. In other words, 85% of the time, coders were certain about selecting Disparate Systems because they chose the same primary and secondary codes. On the other hand, Coordination Breakdown (green) has no overlap with other codes when assigned as the primary code, which shows that coders were absolutely certain in using this code.
Code Wizard automatically generated this information, which helped the team identify the overlapping codes quickly and efficiently, and focus their efforts on the problematic codes. It is important to point out that this matrix (Fig. 7) is not symmetrical. We learned that coders' secondary code assignment depended on the primary code. In other words, when coders selected a well-defined and exclusive code such as Coordination Breakdown (green) as the primary code, they did not experience many difficulties in assigning the same code as the secondary. However, when the primary code was ill defined, such as Content Breakdown (yellow), coders struggled with the secondary code and selected an array of other codes including the well-defined Coordination Breakdown (green).
In addition to generating the aggregated correlations matrix for all coders (Fig. 7), our prototype generated the correlation matrix for each individual coder. The individual coder correlation matrix looks similar to Fig. 7 but uses the data from a single coder. For example, an individual coder could see that she had the most overlap between codes A and C and decide whether she needed to learn more about those codes or bring that up in the team meeting for more discussion. In addition, this coder could compare her coding results with the aggregated results and see how her codes compared with the rest of the team.
This information helped the team to quickly and efficiently find the overlapping codes and address them accordingly. As mentioned earlier, our ICR in the first round of coding was 0.52 so we needed to code a second time to see if taking the above steps had improved our agreement.

Third Challenge: Persistent Low Agreement Rate.
In the second round, we coded each unit of analysis only one time for two main reasons: (1) we had previously discussed the issues of overlapping and ambiguous codes as well as coders' uncertainty, and tested the revised codes in a quick sample coding, (2) all ICR methods merely consider the primary code in their formula, so double coding would not have made a difference at this step, and (3)  After finishing the second round of coding, our prototype merged the code assignments from individual coders and calculated the ICR. Although our ICR had improved from the first round and went up to 0.73 from 0.52, it was still below the acceptable kappa (0.8).
In addition to color-coding, we included the agreement rate per unit of analysis at the end of each row (Fig. 8). This technique helped us quickly locate the disagreements per line and see the degree of disagreement. Based on Fig. 8, some codes still seemed to overlap so we needed to dig deeper to find the extent of code overlaps and the reasons behind the overlaps.
Addressing the Third Challenge: Correlated Disagreement Matrix. Building on the literature, we attempted to address this challenge by leveraging the disagreement in code assignments. Barbour (2001) argues that "the content of disagreement" is more important than the degree of concordance between coders [14]. Lasecki et al. (2014) developed Glance, a tool that measures coders' disagreements to detect problematic or ambiguous analysis queries [30]. Aroyo et al. (2015) indicate that disagreement can address ambiguity, and systematic disagreement among individuals is evidence for having multiple perspectives [31]. Zade et al. (2018) propose metrics to sort and filter disagreements into diversity and divergence to discover the source of disagreements [32].
We took a step further into addressing the code overlaps by focusing on how coders had assigned the codes. We did this by measuring the number of times coders had assigned different codes to a unit of analysis. This is more easily visualized through an example. Imagine three out of six coders assigned code A and three others assigned code C to Unit 1 (Fig. 9). Now, compare Unit 1 to Unit 2, where three coders assigned code B and others assigned A, C, and D (Fig. 9). For Unit 1, we can infer that there may be a strong overlap between codes A and C, especially if we see similar coding patterns repeat for other units of analysis. However, it is difficult to infer much meaningful correlations from the Unit 2 codes. We designed our prototype to detect these patterns of disagreement.
The analogy we used here is similar to our analogy for addressing the second challenge, but unlike the second challenge, here we exclusively focused on the primary code as opposed to comparing it with a secondary code. We merely included the mutual and bilateral relationships between codes (for example correlation between codes B and D) because considering three or higher level of relationships creates complication and confusion computationally and intuitively.

Fig. 9. An example of meaningful patterns in coding
Now, let us describe how our prototype works. Let's imagine five coders (coders 1-5) assigned four codes (A, B, C, and D) to five units of analysis (Fig. 10-a). In this example, Fleiss' kappa is 0.3, which is well below the acceptable level and indicates low agreement between the coders. We designed our prototype to look for potential code overlaps, taking the following steps: (1) The prototype first counts the number of code combinations assigned to each unit of analysis, called "code connection degree," which is determined by counting the minimum number of times a combination of two codes is used in each unit of analysis. For example, in Fig.  10-a, only two codes are used for Unit 1: A and C. A is used one time and C is used four times. Code Wizard counts the number of times these two codes appeared together, looking for connections between AA, CC, and AC.
As can be seen in Fig. 10-b, the code connection degree of AA is 1, CC is 4, and AC is 1 (i.e., minimum number of times that codes A and C are assigned to Unit 1). The code connection degrees between all other codes are zero because they are not used for Unit 1. Below is the formula to calculate the code connection degree: (2) In the second step, the prototype calculates the code connection degree for all units of analysis and provides each code connection in the last row.  (3) Our prototype automatically generates a matrix based on the code connection degrees obtained in the previous step. Since the frequencies of code assignments are not equal and we are interested in discovering the connections between codes, which are not dependent on frequency, we normalized the connection scores (using geometric mean of origin code connection degrees), which we refer to as "correlated disagreement" and calculate it using the following formula: For example, the correlated disagreement coefficient between codes A and C is 0.2.
(4) In the fourth step, the results will be generated and displayed in a matrix, as shown in Fig.  11. Fig. 11-c is the "Correlated Disagreement Matrix"-this matrix represents correlation between mutual codes that is analogous to the correlation matrix in statistics, and it quantifies correlated disagreements among codes.
11-a. Code connection matrix 11-b. Code correlation matrix 11-c. Correlated disagreement matrix Fig. 11. Correlated disagreements between codes of our hypothetical example Below, we list the main properties of our Correlated Disagreement Matrix: 1. It is symmetrical, so we removed its upper half to reduce visual noise. 2. The value of diagonal cells would always be equal to one. Thus, we removed them from the matrix, as they are not informative. 3. In optimal agreement scenarios (ICR= 1.0), all non-diagonal values would be zero. 4. A high value in non-diagonal cells imply two codes were used interchangeably, which is a strong indication of either an overlap or a misinterpretation of the codes. 5. Higher values of non-diagonal cells (closer to 1) means lower agreement rate, and vice versa.
The example matrix in Fig. 11-c shows high disagreement for codes A and B (0.7). This means the team should investigate these two codes.
As mentioned earlier, Code Wizard automatically performs the calculations and generates the matrix, so coders can focus on other critical aspects of collaborative coding that cannot be automated, such as clarification discussions. Now that we have explained the mathematics behind generating the Correlated Disagreement Matrix through an example, let's go back to the coding results of our project. As noted earlier, although our team's ICR score improved from 0.52 to 0.73, it was still below the acceptable threshold (0.8). The Correlated Disagreement Matrix generated for our second round of coding (Fig. 12) immediately revealed the problematic areas: coders had the highest correlated disagreement coefficient between Disparate Systems and Computational Mechanism Issue, as well as Content Breakdown and Source Breakdown. Therefore, our team got together to discuss these codes and further revised them.
After team discussions to address ambiguous and overlapping codes and to make our codes exclusive, our team performed a third round of coding to see how our agreement rate changed after taking the above steps.
Our ICR in the third round of coding was 0.81, which is above the acceptable threshold. Our team was satisfied with the agreement rate so we did not hold a meeting to discuss further disagreements. However, Code Wizard generated the Correlated Disagreement Matrix (Fig. 13) to show what changes led to a higher agreement rate.  As can be seen in Fig. 13, the number of cells with zero value increased, which indicates the exclusivity of those codes. In other words, the problem of high correlation between many codes is addressed here. As Fig. 12 illustrates, in the second round of coding, Paper Mechanism Issues (orange) correlated with both Content Breakdown (yellow) and Coordination Breakdown (green). However, in the third round of coding (Fig. 13), there is no longer any correlation between these codes. In addition, the correlation rate for many codes is reduced, such as for Source Breakdown and Content Breakdown.
In our analysis, we did not add, remove, or alter codes but because Code Wizard is developed in Excel, codes can easily be added, removed, or altered if needed. We programmed Code Wizard to automatically update the codes based on changes in the codebook or where needed.

DISCUSSION
This study describes the development of a prototype for a coding tool called Code Wizard with the goal of enabling and supporting efficient collaborative coding using visualization techniques and colors.
We discussed three main challenges we encountered throughout the process of collaborative coding that were also expressed by many other scholars, as described in Section 2. We illustrated how Code Wizard helped the process of addressing each challenge and improving our team's collaboration.
Software packages such as NVivo, Dedoose, and Atlas.ti support collaborative coding for qualitative analysis at some levels. In NVivo, Dedoose, and Atlas.ti coding stripes show how different coders coded a piece of data [18, 20, and 21]. In NVivo, the coding comparison query feature enables coders to compare coding of two users or two groups of users and calculates Cohen's kappa and direct agreement percentage between them. Dedoose calculates Cohen's kappa and Pearson's correlation coefficient between two users. However, NVivo and Dedoose do not currently evaluate the ICR coefficient between more than two coders. Atlas.ti does not calculate any type of ICR coefficient. Also, NVivo, Dedoose, and Atlas.ti present the cooccurrence matrix of codes selected for units of analysis simultaneously. NVivo can produce the co-occurrence matrix for codes that assigned by two coders for similar units of analysis too. However, NVivo, Dedoose, and Atlas.ti fall short in creating co-occurrence matrices if the number of coders exceeds two. Code Wizard produces the Correlated Disagreement Matrix, which is conceptually related to the co-occurrence matrix, for any number of coders following the approach presented in Section 3.3.
Code Wizard was gradually developed during the collaborative coding process, so we used the coders' concerns and feedback to improve the tool. For example, the idea of coding a unit of analysis twice came from the coders' discussions in meetings. During the discussion sessions, we realized that coders had difficulties picking one code to assign to the units of analysis. Therefore, we added a feature to the tool to allow them assign two codes that fit best. Having two codes was challenging because ICR coefficients are calculated based on one code. Although this feature would not change the ICR coefficient directly, allowing coders to select two codes and embracing and presenting the uncertainties during the discussion sessions helped the team collaboration.
Code Wizard supported team collaboration in two main areas, the process of coding and the practice of coding. We organized our findings according to these two categories.

Process of Coding
This category includes the mechanisms our teams used to conduct qualitative coding, select a coding tool, perform the coding, measure ICR, and similar activities.

Collaboration Evolved Over Time.
Code Wizard engaged the team in every step of the process of coding, discussing, and revising the codebook. As the team continually shaped the codebook, our team's collaboration formed around our understanding of the phenomena, which led to further shaping the codebook to create exclusive codes for our research. As Weston et al. indicate (2001), "team builds codes and coding builds a team through the creation of shared interpretation and understanding of the phenomenon being studied" [33]. Code Wizard helped a very diverse team with little experience in the area of disaster management learn and collaborate throughout the coding process in a meaningful way because they could see the impact of their work on the entire process as well as on the research results.
We acknowledge that as the coding team becomes more familiar with the codes and more accomplished at understanding the data and applying the codes, they become faster and better at coding, and potentially their level of engagement can increase. But we should recognize that the coding team changed every 10-12 weeks. While we did not completely control for the new coders, we realized that Code Wizard made onboarding and engaging the new coders quick and easy and the new coders were as engaged as the people who had been on board from the beginning.

Code Wizard Made the Coding Process Visible.
Collaborative coding includes multiple steps and tasks that are often not talked about or are easy to overlook in qualitative studies. Transparency of the qualitative research process including how the decisions were made and what factors were involved in reaching an agreement are often not discussed in qualitative papers and tend to get buried under reporting the final results. Yet, each of these steps plays a crucial role in ensuring the validity of research findings and ensuring that the codes are not superficial. As Ryan indicates, "it is important to make visible the researcher's decisions and processes. The intricate processes of organising, coding and analysing the data are often rendered invisible in the presentation of the research findings, which requires a 'leap of faith' for the reader." Ryan adds that computer tools can help "make the research process more transparent, without sacrificing rich, interpretive analysis by the researcher" [34]. We programmed Code Wizard to automatically record key decisions, reflections, variations, and emergent ideas for each round of coding in a new spreadsheet. This was possible by comparing visualizations generated from collaborative coding in different rounds. Team members could also see where other coders were in the process, each individual coder's struggles, and how their uncertainties could have a significant impact in the coding process and discussions.

Coders' Commitment to Data as Opposed to Data Alienation.
Our team's experience diverged from the criticism that using computer-assisted coding programs would alienate researchers from the data [35]. Code Wizard helped the team engage with data in three ways: (1) it made it easy for coders to see in what ways even small changes to the codebook can make a significant impact on the result, (2) it made the coding process transparent so the coders could see where they needed to focus, as opposed to having lengthy and inefficient meetings, which is what we initially experienced with using Dedoose, and (3) Code Wizard provided real-time results of the changes in coding, so coders could interact with data in a more dynamic way. 4 individual coders' sense of authority and responsibility, which led to more engagement in the process. In team discussions, individual coders would confidently share new ideas, offer feedback, and talk about improving the results.
4.1.5 Eliminating the Need for Backtracking the Codes Line by Line. Code Wizard helped the process of collaborative coding by eliminating the need for going over the units of analyses line by line to find the disagreements. Code Wizard identified and located the primary disagreements so the discussion would be focused on those specific disagreements. Backtracking to locate disagreements can take months to finish. In addition, by the time coders reviewed a few disagreements, they could forget the details of earlier conversations and the earlier reasoning. At times, the clarification discussions would lead to changing the codes or revising the definitions and that would make earlier conversations inapplicable.

Practice of Coding
This category includes how the team communicated and exchanged information at every step in the process.

Using Uncertainties to Make Communications More Explicit.
We identified certainty/uncertainty as an important factor in the practice of collaborative coding. We measured certainty by asking coders to code each unit of analysis two times in the first coding round. Measuring uncertainty helped team members to not only find ambiguous codes and code overlaps, but also to find the extent to which those codes overlapped. This information helped the team exchange their understanding of the codes and context, and develop actionable plans accordingly.

Accelerating Coding through Use of Colors and Visualizations.
Using colors proved to be invaluable in locating disagreements and it made discussions quicker and more efficient. We used colors in two main ways: (1) we used distinct colors for codes, which helped in locating the disagreements quickly, and (2) we used shades of red to show the degree of disagreements. Dark red represented significant disagreement, and light pink was used for trivial disagreement. While using colors was desirable by coders and accelerated the coding process, it also created some issues for the team. Despite our best effort to assign distinctive and high-contrast colors to different codes, we realized that colors would appear differently on different computers or operating systems. This unforeseen shift in colors created confusion because at times coders would see different codes being represented by relatively similar colors. In addition, coders initially had difficulties understanding the correlation matrices and visualizations (Figs. 7, 11, and 12). We partially resolved this issue by refining and simplifying the visualizations. We are also experimenting with other types of visualizations in future studies.

Interacting with Real-Time Results.
Code Wizard automatically and simultaneously generated all the visualizations and applied colors. This feature tremendously helped team discussions because it enabled the team to quickly and easily make changes and see the results in real-time during discussion sessions. The coders could individually or in a group modify their codes in the master Excel sheet and instantly see the results and updated visualizations. Unlike tools such as NVivo, in Code Wizard we did not need to upload a new file if people changed their codes during discussion sessions. One of the experienced researchers on the team wrote, "[Code Wizard] facilitated the reconciliation process by allowing all coders to simultaneously see all experts and each coder's input, making it easy to discuss discrepancies."

Code Wizard Promoted Diverse Team Integration.
In our project, we worked with a diverse group of people with no experience in emergency management who came from different disciplines (civil engineering, human-centered design & engineering, electrical engineering, and cognitive psychology). Working with a diverse team introduced some challenges in reaching a common language that was understandable by all. Thus, we paid special attention to creating an inclusive and accessible tool that respects differences among a diverse group of coders and supports their comfort level.
4.2.5 Little Training Required. Programming Code Wizard's prototype within Excel, a familiar software, meant not much training was required. Team members sent the individual coding to the team lead, who incorporated them into Code Wizard. The rest of the processes are all automated. During discussion sessions, coders can modify their coding in Code Wizard and the result will change automatically in real-time.
As noted earlier, we had initially experienced many difficulties with existing commercial coding tools. We spent more time learning their features than working on our research. Our team expressed feeling relief when Code Wizard was being developed during the course of the research. One of our team members programmed Excel as a platform and this decision was highly supported by the team members because (1) everyone was familiar with Excel, (b) everyone had access to Excel, (c) Excel was free for students-if they did not already have it installed it on their computers, it was available through the university IT center.

LIMITATIONS
In this work, we focused on improving collaborative coding experience in qualitative research. Our case study showed promising results when multiple coders collaborate on a large dataset made of textual field notes in an inductive qualitative research. Further research will show whether Code Wizard would support other types of qualitative data such as audio, tweets, or other forms of text media. We also do not know the possible benefits or impacts of using Code Wizard on larger teams or teams of two coders.
Code Wizard may not be as helpful when a gold standard coding scheme is already established and when the codes are mutually exclusive, but it may help train the coders. In addition, Code Wizard currently does not resolve the issue of measuring ICR for multiple coding and hierarchical coding, but we look forward to extending Code Wizard to handle multiple coding and hierarchical coding.
In our example, we used Fleiss' kappa, which does not work when no code is assigned to a unit of analysis, but we would like to test Code Wizard for such instances, as either primary or secondary. Finally, we observed that the coders sometimes struggled to understand the visualizations presented in Figs. 7, 11, and 12. We revised these visualizations to make them as simple as possible. We did not test Code Wizard with other types of visualizations such as chord diagram, but we plan to do in future studies. As we improve Code Wizard, we would like to explore ways of extending it to support other types of coding and incorporate more effective visualization techniques.

CONCLUSIONS
This research is a contribution to an ongoing effort to address the challenges of collaborative coding of qualitative data using computerized tools. We believe that there is a need for a simpler

132:23
Proceedings of the ACM on Human-Computer Interaction, Vol. 2, No. CSCW, Article 132, Publication date: November 2018. tool to help teams that do not require the heavy weight and sophisticated features offered by commercial tools. Completely unaided collaborative coding is difficult, while commercial tools can add overhead. We should make the distinction between the support offered by a simple and lightweight tool such as Code Wizard versus the support that the heavy weight commercial tools offer. Our goal was to design a middle-level tool for teams with simple coding needs and not to replace the existing tools. We believe there is space to design tools for this middle-level coding, and we hope our effort can open a new avenue for the CSCW community interested in developing simple tools to support collaborative coding.
As qualitative data analysis becomes increasingly important in various fields and with advances in technology, we now need to process data in larger amounts and with more complexity. We believe that visual analytics and computerized coding tools can support the tedious process of collaborative coding more efficiently. Our work simplified the process of detecting ambiguous and overlapping codes and offered insights into the process of measuring ICR. If we combine all the solutions described in this paper, Code Wizard would be able to more methodically manage the collaboration.