{'fraction_non_alphanumeric': 0.02183558829315206, 'fraction_numerical': 0.005420646543244956, 'mean_word_length': 5.1527703066164605, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Can Microtask Programming Work in Industry?

A critical issue in software development projects in IT service companies is finding the right people at the right time. By enabling assignments of tasks to people to be more fluid, the use of crowdsourcing approaches within a company offers a potential solution to this challenge. Inside a company, as multiple system development projects are ongoing separately, developers with slack time on one project might use this time to contribute to other projects. In this paper, we report on a case study of the application of crowdsourcing within an industrial web application system development project in a large telecommunications company. Developers worked with system specifications which were organized into a set of microtasks, offering a set of short and self-contained descriptions. When crowd workers in other projects had slack time, they fetched and completed microtasks. Our results offer initial evidence for the potential value of microtask programming in increasing the fluidity of team assignments within a company. Crowd contributors to the project were able to onboard and contribute to a new project in less than 2 hours. After onboarding, the crowd workers were together able to successfully implement a small program which contained only a small number of defects. Interview and survey data gathered from project participants revealed that crowd workers reported that they perceived onboarding costs to be reduced and did not experience issues with the reduced face to face communication, but experienced challenges with motivation.


INTRODUCTION
A critical issue in software development projects in IT service companies is finding developers with the right knowledge. For example, one report suggests there will be a shortage of 430,000 IT specialists in 2025 in Japan [22]. When projects require more resources than available, it is often challenging to quickly meet these needs. Recruiting new employees from outside a company requires a substantial investment of time and cost. Developers might instead be brought in from other ongoing system development projects or teams inside the company. However, even for developers already working within a company, it can require a substantial investment of time and effort to successfully onboard them onto a project. For example, developers need to learn project background knowledge such as the system architecture, project configuration, and coding conventions. For these reasons, balancing resources within a company often does not work effectively [3,4,8,9].
One potential solution to these challenges is the use of crowdsourcing [2,[18][19][20]. In crowdsourced software engineering, work traditionally done inside a company is outsourced to an undefined crowd in the form of an open call [21]. Successful commercial platforms for crowdsourcing software engineering work include TopCoder, AppStori, uTest, and TestFlight. Crowdsourcing offers companies the potential for fluidity, enabling workers from outside to be recruited on demand at the moment in which work must be completed. One form of crowdsourcing is microtask programming, where programming tasks are decomposed into short, self-contained tasks with a clear objective [20]. By decomposing the specifications of a system into small, self-contained descriptions, microtasking aims to further decontextualize work, enabling even greater fluidity by reducing the needed context to onboard.
While crowdsourcing, by definition, involves recruiting contributors from outside a company or organization, it might also be possible to apply crowdsourcing within a company. Rather than offer work to external developers, developers working on other projects within a company with available slack time might use this time to complete microtasks and contribute to another project. For companies with closed-source code and confidential information and intellectual property to protect, this model offers many of the potential crowdsourcing benefits of lower onboarding costs and greater resource fluidity with fewer of the potential drawbacks.
In this paper, we report on a project at NTT which applied microtask programming to an industrial web application development project. Inspired by behavior-driven microtask programming [2], a workflow was used where crowd workers could make two types of microtask contributions: 1) implement a micro-specification or 2) review an implemented micro-specification. Crowd workers followed Test Driven Development (TDD) to implement micro-specifications. Micro-specifications were defined and collected in a ticket pool, and developers with slack time on their own project were able to fetch and complete microtasks. Inspired by the role of the co-pilot in TopCoder [26] responsible for managing and overseeing development work, the work of decomposing and integrating microtasks was done by two engineers dedicated to the project while six internal crowd workers completed microtasks ( Fig. 1). The dedicated software engineer served three roles in the project, working as a software designer to create microtasks, as a developer to implement several tasks, and as a tester.
Overall, the project achieved its objectives and a web application for managing finance closing processes was successfully created. Crowd workers together implemented approximately 2800 LOC. All were able to initially onboard onto the project in less than 2 hours and successfully communicated via an issue tracking system (ITS). Crowd workers reported that they perceived onboarding costs to be reduced and did not experience issues with the reduced face-to-face communication, but did experience challenges with motivation.
The remainder of this paper is organized as follows. Section 2 describes related work in crowdsourced software development, microtask programming, and inner sourcing. Section 3 describes the microtask programming process and tools adopted in the case study. Section 4 first presents results on the outcome of the project, including onboarding activities, and the describes the perceptions of the project participants about the use of a microtask programming process inside a company. Section V discusses the implications of our findings, and Section VI concludes.

RELATED WORK
A variety of work has explored crowdsourcing approaches for software development activities such as design [16], implementation [2,6,15,18], and testing [7]. Companies such as TopCoder, uTest, and UserTesting.com offer a platform and community for crowdsourcing software activities to an external crowd. A study of TopCoder found a number of factors that impact project quality, including the number of contemporary projects, the length of documents, and the number of registered developers [26].
When developers join a new project, they may face a number of onboarding barriers, including installing necessary tools, identifying and downloading dependencies, and configuring their build environment, understanding the codebase, and identifying a task [14,24,27]. As a result, successfully onboarding onto a new project may require weeks of time, creating a substantial barrier to fluidly assigning developers to match project resource needs.
Crowdsourced programming environments have been designed to reduce some of these barriers by offering a preconfigured programming environment. In Apparition, developers create microtasks for crowd workers to build small user interface elements and their behaviors, which crowd workers can then complete in a dedicated environment [15]. In CodeOn, developers speak requests for small contributions which other developers can then supply [6]. In microtask programming, developers complete short microtasks in a dedicated environment in which they are given a function or test and are asked to make a small change to it [1,2,18]. By decontextualizing work from the larger project context and offering a dedicated environment with the necessary background information and editors to make the contribution, developers are able to make contributions in under 5 minutes with only a short onboarding period. Studies of the use of crowdsourcing in industry have found that companies have relatively low awareness of these new approaches [23].
A key challenge in crowdsourced software development is in designing effective coordination mechanisms, which may have a variety of benefits and drawbacks [10,11]. One approach is for coordination between the requirements engineer and crowd workers to occur entirely through the specifications themselves [2]. An alternative approach is for the requirements engineer or developer responsible for creating the microtasks to directly coordinate with each worker [15]. Another key choice is the ways in which crowd workers themselves may coordinate, such as through instant messaging [2] or through more structured interactions such as around issues [17]. Crowd workers themselves have expressed a desire for direct worker-to-worker communication when handoffs or reviews necessitate interactions between workers [17].
Noting the success of open source approaches, companies have explored adopting open source software development practices inside their organizations through a set of practices named inner sourcing [5]. Developers working using inner sourcing do not belong to a single project, and anybody in the organization may contribute to all projects in it [25]. For companies building closedsource software with confidential information and intellectual property, inner source offers new techniques and models for encouraging internal collaboration by applying open-source best practices within organizations [12,13]. Studies examining the impact of inner sourcing found potential benefits such as increased development efficiency, higher code quality, and quicker development cycles [5].
Our work builds on these existing techniques and studies, focusing on the potential fluidity offered by adopting microtasked development in an industrial context.  Figure 2: The project consisted of a frontend and backend with 6 major components. In both the frontend and backend, components which required more project knowledge were implemented by the dedicated software engineer (gray background) while those which required less project knowledge were implemented by the crowd workers (white background).

MICROTASK PROGRAMMING IN A COMPANY 3.1 Project Overview
In the project, an in-house web application system for managing information on finance closing processes was built. The existing system had been used by an organization within NTT, had about 50 users, and had operated for five years. Due to business reasons including the expiration of a license, it was decided to reimplement the system. In order to more fluidly assign resources to the project, the project was organized using a crowdsourced software development approach.

Organization Structure
All participants in the project were employees of NTT. As shown in Figure 1, two dedicated engineers were assigned to work full time on the project, a requirements engineer and a software engineer. In addition, 6 crowd workers, who were primarily assigned to other projects inside the company, were asked to make use of their available slack time to contribute to the project. For example, if they was not busy at the end of the day or in the morning, they might use this time to contribute. The dedicated engineers and crowd workers were located in geographically distributed areas, with the dedicated engineers located in a single site and the crowd workers located at a different site.

System Architecture
Parts that required more project knowledge were assigned to the dedicated software engineer, while those which required less were implemented by the crowd workers ( Figure 2). The crowd workers implemented two parts of the web application (UI Components and Functions API), and the dedicated software engineer implemented three (Controllers, Stores, and Persistence API). Implementing these three required more knowledge of the data model and its complexities. The web application was implemented using React and Redux in the frontend and Express.js and MySQL in the backend.
In the frontend, the dedicated software engineer implemented the Controllers and Stores, and crowd workers implemented the UI Components. The UI Components were first decomposed into micro-specifications by the dedicated software engineer, and the crowd workers then implemented them. The micro-specifications in the frontend did not have mutual dependencies, and there was not a specific order in which they needed to be completed.
In the backend, the dedicated software engineer implemented the Persistence API, and the crowd workers implemented the Functions API. The Persistence and Functions API did not have dependencies. The dedicated software engineer generated several micro-specifications for the Functions API, which had mutual dependencies. This required managing the order in which they were implemented. Figure 3 depicts the overall development process used and the artifacts generated at each step. The development process consisted of four steps: basic design, detailed design, implementation and unit testing, and system testing. Basic design was completed by the requirements engineer in the traditional way. The requirements engineer created four artifacts: a conceptual data model, use cases, screen transitions, and a basic UI design.

Software Development Process
Next, in the detailed design step, the Functions API and UI components specifications were decomposed by the dedicated software engineer into a set of micro-specifications. For example, in Figure 4, the Function API specification was decomposed into three microspecifications. The dedicated software engineer generated three types of artifacts: abstract data types, Functions API (backend), and UI components micro-specifications(frontend).
In the implementation and unit testing step, each of the microspecifications were then implemented by the crowd workers. Crowd workers followed Test Driven Development (TDD) to implement micro-specifications. For example, Figure 5 depicts an example of a micro-specification being fetched, implemented, and pushed by one crowd worker and reviewed by a second crowd worker.
In the testing step, the dedicated software engineer created a test suite. They then tried to build all of the components together. There were several defects that were found. In some cases, defects were fixed by the dedicated software engineer, while in others they were fixed by the crowd workers.  Update the checksheet with input data. 1. convert from checksheetData parameters to array of DBChecksheetUpdateInfo, call 3rd party API updateObjectImplementation, and return "0".

Workflow and Tools for Crowd Workers
create "DBChecksheetUpdateInfo" from each "checksheetData/checksheetItems", and push to array.
set parameter checksheetData/checksheetId tochecksheetId of DBChecksheetUpdateInfo 2. valid parameters as follow, if there is invalid parameter then Set "the parameter is invalid"to error message and throw the error.
parameter checksheetData is null parameter checksheetData/checksheetId is null paramter checksheetData/checksheetItems is null or empty paramter checksheetData/checksheetItems/checksheetItemId is null paramter checksheetData/checksheetItems/status is null or not any of "0", "1", "2". 3. If 1st step result is null, then set error message is "faild update"and throw Error Figure 4: The function updateFinChecksheet API was described through three micro-specifications created by the dedicated software engineer. Each micro-specifications was then mapped to a microtask. A crowd worker completed each microtask, and a second crowd worker reviewed the work.  Figure 5: An example of work being completed by two crowd workers within the microtask programming workflow. The dedicated software engineer first decomposed each specification into multiple micro-specifications, creating a ticket describing each micro-specification to implement. Crowd worker one may then fetch a ticket from the issue tracking system (ITS), complete the work, push the source code, and create a pull request in the version control system. Crowd worker two may continue the work, fetching the pull request and reviewing the source code implemented by crowd worker one. If crowd worker two approves the work, they may then merge the pull request into the version control system (VCS).

Microspecifications
committed. Two key tools were used: an Issue Tracking System (ITS) and Version Control System (VCS). Participants used the ITS  Figure 6: In the microtask programming workflow, each crowd worker worked through a series of steps. Workers first fetched a ticket, which included backend and frontend tickets as well as review tasks. For implementation tasks, workers then completed several programming steps before committing their work and finishing the microtask.
to manage the status of the work and the version control system to manage changes to code.
In the microtask programming workflow, the software engineer first decomposed the specifications generated by the requirements engineer into micro-specifications and issued a ticket for each micro-specification in the ITS (an example of a micro-specification is shown in Figure.4). Next, a crowd worker fetched a ticket from the ITS. Crowd workers selected the work that they chose to do. They then implemented the micro-specification described in it by using the template imported from the VCS. After completing the implementation, the crowd worker then pushed the source code and created a pull request. A second crowd worker then reviewed the contribution. After fetching the ticket in the form of a pull request, the second worker reviewed the source code. If they approved the work, they then merged the pull request.
Crowd workers also made use of the ITS to communicate project knowledge. Crowd workers asked the dedicated software engineer questions about the micro-specifications posted on the ITS. If the designer found the question to be broadly applicable, they shared it with all of the crowd workers by posting it to an internal wiki. When crowd workers began contributing to the project, they were then asked to first read the internal wiki.

RESULTS
In the following section, we first report on the outcome of the project, describing the activities that occurred and the results of the project. We then report data on the productivity of crowd workers within the project. Finally, we examine the project participant's perceptions of the suitability and effectiveness of applying microtask programming in a company setting through collected interview and survey data.

Project Activities
Work in the project occurred in five phases over a seven week period: basic design (one week), detailed design (one week), sprint 1 (three weeks implementation and unit testing, and three-day system testing), and sprint 2 (one week implementation and unit testing, and two days system testing). Transient crowd workers contributed to the project in both sprints. The dedicated software engineer was the same in the two sprints. In each sprint, crowd workers were assigned to either frontend or backend work.
To onboard onto the project, crowd workers attended a kickoff meeting, read content on a dedicated wiki, and set up their computer environment. Before beginning implementation work, all of the crowd workers attended a 30-minute kickoff meeting in which the requirements engineer and the dedicated software engineer offered an overview of the microtask programming workflow. Rather than impart knowledge about the project itself, the briefing focused exclusively on the use microtask programming. After the kickoff meeting, when the crowd workers were ready to begin working on the tasks, they setup their environment. To do so, they read the information in the wiki which we had prepared beforehand. The wiki instructed them about how to set up the environment and how to use it for work. Rather than use a pre-defined environment within a virtual machine, crowd workers instead set up their own laptop or desktop computers from scratch. Crowd workers reported that it took approximately two hours to set up their environment on average, including reading and understanding the contents of the wiki and configuring their computerâĂŹs environment. The environment consisted of the Visual Studio IDE, web browser, Git VCS, node.js and some JavaScript libraries. Crowd workers spent approximately 30 to 60 minutes configuring their computer environment.
After the crowd workers completed their work in the implementation step in sprints 1 and 2, the dedicated engineer tested the project. The contributions from each microtask were composed into an assembled program and tested through a test suite to evaluate if the system satisfied its requirements. System testing was conducted using standard company practices, where test cases were implemented based on the requirements described in the use case and screen transition artifacts. Two types of defects were found. Some defects were related to the micro-specifications created by the dedicated software engineer. Other defects were related to the implementation of the micro-specifications by the crowd. The dedicated software engineer fixed all of the defects.
All of the defects uncovered were minor issues. The dedicated software engineer confirmed that the specifications were completely implemented. Based on the successful test results, the project entered the next phase of user acceptance testing.

Crowd Worker Productivity
Across the two sprints, the crowd workers implemented 9 Functions API and 5 UI Components through 50 microtasks. The work completed is listed in Table. 1. In sprint 1, five crowd workers worked on the backend to develop five functions. On average, each crowd worker implemented 4 microtasks in the first sprint. Three crowd workers worked on the frontend to develop three UI Components based on seven microtasks created. Two crowd workers worked  on both the frontend and backend. In sprint 2, three crowd workers worked on the backend to develop four functions. Each crowd worker implemented an average of 6 microtasks. Two crowd workers worked on the frontend to develop two UI Components based on five microtasks. In contrast to sprint 1, each crowd worker worked exclusively on either the frontend or the backend. While seven crowd workers were initially invited, one crowd worker was not able to contribute to the project as they did not have any slack time.
To assess the output of the crowd and dedicated software engineer, we measured the lines of code (LOC) produced in each sprint. Table 2 summarizes the results. In sprint 1, the crowd workers and the dedicated software engineer developed 1,591 LOC and 2,282 LOC, respectively. In sprint 2, the crowd workers developed 1,300 LOC and the dedicated software engineer developed 2,775 LOC. The dedicated software engineer implemented more LOC in this sprint as they implemented templates and packages in the frontend. The final project consisted of approximately 8,000 LOC, of which about 2,900 LOC (36%) were implemented by the crowd workers. The average implementation microtask involved 58 lines of code .
We also used the timestamp data recorded in the tickets of the issue tracking system to examine the productivity of the crowd workers. Table. 3 lists the average working time and the number of fetched tickets, and micro-specification for each crowd worker. Frontend tasks took more time than backend tasks. In total, 50 microtasks were generated and 66 were submitted by crowd workers. Microtasks may be submitted more than once when a pull request was rejected. In total, 50 microtasks were generated, and crowd workers submitted 66 microtasks. On average, 75% of implementation microtasks were accepted. There are two reasons why average times were in some cases more than one hour. First, some tasks were too big or too complex to do in an hour. The second is that crowd workers sometimes fetched a ticket and switched to another task. Ticket timestamps sometimes indicated that the crowd worker fetched tasks in the evening and did not start working on it until the following day. On several occasions, one crowd worker fetched a task and only began it later. As a result, Crowd Worker Two in the backend had an average work time of 11hr, 35 min. While workers had the ability to pause the time in the ITS, some forgot to do so. Figure 7 depicts the work done by each crowd worker, from the time they fetched a microtask until they submitted it. For example, on Backend artifact 3, Crowd Worker 6 first completed a Review microtask in about one hour, Crowd Worker 2 completed an implementation microtask in about one hour, Crowd Worker six then completed Review and Implementation microtasks in succession, and finally Crowd Worker 2 completed a a Review microtask.
Crowd workers were prohibited from fetching multiple microtask at the same time, although they sometimes forgot to close the ticket and started a new task. Work on an individual artifact often switched between different crowd workers. Work often clustered around points of time with more intense activity. Most crowd workers contributed during the daytime or working for 2 or 3 hours in the evening.
We did not precisely record the hours worked on the project by the dedicated requirements engineer and software engineer (Fig. 3). However, we were able to estimate the time. On average, the dedicated engineers worked for 8 hours per day. The requirements engineer spent one week on the project (40 hours). The software engineer spent 40 hours in the detailed design phase. In those 40 hours, the software engineer exclusively worked on converting basic design diagrams to function specifications and preparing microtasks. In the last phase, the software engineer spent an additional 40 hours conducting system testing. Figure 7: A visualization of crowd workers' switching between artifacts during the project. The vertical axis is the artifact workers were contributing to and the horizontal axis time. Each bar in the plot corresponds to a single microtask (either implementation or review microtask). Crowd workers are indicated through colors. The bars' pattern indicates the task type, with striped bars indicating implementation tasks and filled bars indicating review tasks.

Perceptions of Use by Crowd Workers
To understand the perceptions of the crowd workers about the suitability and value of microtask programming in a company setting, we gathered data through two methods. One week after the completion of the project, we asked the 6 crowd workers to complete a short questionnaire about their experiences with microtask programming inside a company. The questions focused on onboarding challenges, the granularity of microtasks, the freedom to choose a task, the motivation of working in the microtask programming approach, and communication among workers. We then later conducted 15-30 minute semi-structured interviews with each of the six crowd workers. The open-ended questions focused on onboarding challenges, the granularity of microtasks, the freedom to choose a task, the motivation of working in the microtask programming approach, and communication among workers.

Questionnaire Results.
The results from the questionnaire are listed in Figure 8. The crowd workers mostly agreed that microtask programming reduced the time required to onboard onto a project. Each crowd worker took on average only 150 minutes (30 minutes for the kickoff meeting and 120 configuring the environment) to join the project. As much of this time was dedicated to understanding the microtask workflow and configuring the related tools, this time would likely be less were crowd workers to participate in another microtask programming project.
Workers reported varied opinions on working with microtasks. Most agreed with ability to select or skip each microtask. However, crowd workers reported that staying motivated was often hard. Crowd workers did not find the lack of face to face communication to be challenging.

Interview
Results. Most of the crowd workers felt that microtask programming decreased onboarding costs. One reported that the wiki which had been prepared beforehand effectively reduced the onboarding cost by including detailed information on how to set up the environments and how to use it for work. Other workers reported difficulties onboarding. One reported that it takes time to learn rules and appropriate methods of work. Another worker reported that a pre-defined environment such as a virtual machine would have considerably reduced onboarding costs.
Most workers had neutral opinions about the clarity of the microtasks. For three workers, some micro-specifications were not clear. For example, one reported that: "Text-based specification is hard to understand. If the specification had included figures and/or models, it would have been better. " Workers reported following two approaches for resolving unclear micro-specifications. The first one was skipping the ticket in the ITS and beginning work on another ticket. The second approach was to create a ticket in the ITS to ask the dedicated software engineer a question. Crowd workers were comfortable selecting microtasks themselves. Two workers reported that micro-specifications had many details about the logic that were helpful. Two workers reported that implementation microtasks were a good fit for the backend, as they were neither too small nor too big. Workers also reported concerns about implementation microtasks.
One worker reported concern that the sequence of microtasks was not specified, as how a micro-specification was implemented might impact the implementation of another micro-specification. Another worker reported being concerned that a behavior implemented earlier might no longer work correctly after another microspecification was implemented. When a crowd worker implemented a micro-specification and wrote a test for it, they focused only on the micro-specification they were assigned. When they wrote the test, it might pass. But it might later fail after another crowd worker implemented other functionality. For example, Crowd Worker 1 wrote a test for micro-specification A where he invoked a function with empty parameters. At the time, the test passed. However, according to micro-specification B, the function must be invoked with parameters. When Crowd Worker 2 implemented this behavior, the test for micro-specification A then failed. In this way, unit tests helped to facilitate coordination.
Crowd workers had different views about the value of review microtasks. Most thought review microtasks were easy to understand, as the scope was narrow. However, when the micro-specification was incorrect or unclear, this created challenges. To resolve this, crowd workers had to write on the ticket to communicate about the specification. They instead wished to be able to communicate directly with the dedicated software engineer who had authored the micro-specification. Moreover, one worker reported that, while he also does review tasks in regular software development, review microtasks are different. He reported that: "Usually, the viewpoints of [the] review are based on syntax such as coding rules, format, and so on. But in this project, I had to check whether the code meets behavior; it takes a longer time than usual. " Another crowd worker complained that his work was rejected several times because the code violated code rules.
All crowd workers reported that understanding a micro-specification and implementing backend microtasks could be completed in less than one hour, with a wide range of responses ranging from 20 to 60 minutes. They reported that completing frontend microtasks might take more than one hour, as the microtasks were larger.
Crowd workers were transient, and they could work on the project in their slack time. One of the workers reported a preference to complete multiple microtasks at a time. He told us he prefers to complete 5 or 6 microtasks in one session for a couple of hours.
Based on the results of the questionnaire, we investigated the challenges for crowd workers in staying motivated in microtask programming. Crowd workers reported several reasons. Two reported that they did not feel that they belonged to the project. These crowd workers reported that if a microtask had been assigned to them and the microtask had a due date, they would have been more motivated. Two other workers reported that it was not clear if there remained incomplete microtasks. If someone had asked them to complete the incomplete microtasks which, they would have done that. Another reported that since he thought someone else would complete microtasks, he did not feel that he should do that.
Crowd workers reported several suggestions for increasing the project velocity and the motivation of crowd workers. Two workers suggested that instead of arbitrarily choosing microtasks they should instead be assigned. Workers would then know that they had a microtask that they must complete before its due date. Three crowd workers shared that if there was an incentive like a score or money, they would work harder. Another crowd worker suggested dedicating a specific day to microtask programming.
The interview results confirm many of the questionnaire results. Most crowd workers did not have problems with the lack of faceto-face meetings with crowd workers. Four workers reported that they only needed communication in the onboarding phase of the project or their first microtasks. After that, they did not have specific questions. Thus they felt they did not need any specific tools for communication. They believed that workflow and tasks were clear enough. On the other hand, two participants reported it would be better if all communication were in a shared place like a question and answer tool. They felt that sharing information might increase their productivity. One participant reported: "... I prefer chat to [a] ticket. I feel that it is a good point to record the history of the actor's actions using tickets. However, in many scenes, I need quick answers from software designers. So, the chat is better than a ticket. "

Perceptions of Use by Dedicated Software Engineer
We conducted an interview with the dedicated software engineer to gather feedback about the challenges he faced when he decomposed tasks into microtasks and when he assembled the contributions into a finished application. Although the software engineer could complete all of the tasks by himself, the software engineer reported that his time commitment was reduced. While crowd workers were working on the implementation microtasks, the software engineer used that time to prepare the integration test and micro-specification for the next sprint. As he had designed the project, he was able to complete the tasks in a shorter time than the crowd workers. It is not clear if someone else had created the tasks and the software engineer was not involved in the design phase how much it time it would have taken for the dedicated software engineer to complete all of the tasks.
Manually decomposing tasks into microtasks was challenging. In the backend, he tried to decompose a specification of a module into a set of testable, smallest behaviors. It was challenging to describe complex behaviors using only text. In the frontend, he tried to decompose the web pages into a set of components. Each was the smallest meaningful unit of the UI element. It was challenging to consider what the minimum information needed to implement a microtask was. The software engineer struggled to decide the level of detail necessary for the micro-specifications. It was difficult for the software engineer to predict how much time the crowd workers would spend understanding the micro-specifications he wrote. He also thought that the reason why the tasks took much more time than he expected was the size of the microtasks. He described much of the contents in the UI componentâĂŹs specifications. So these required much time to implement.
Aggregating microtasks and conducting integration tests was not straightforward for the software engineer. There were not automated continuous integration tools, which added much time. He suggested that the test results might be used to monitor project progress.

LIMITATIONS AND THREATS TO VALIDITY
When we design any case study, care should be taken to mitigate threats to validity [28].
Construct validity addresses the degree to which a case study aligns with the theoretical concepts used. To reinforce construct validity, there are three ways: using multiple sources of reliable evidence, establishing a chain of evidence, and having key informants review reports of a draft case study [28]. In this case study, we used only one source from one web application system development project. To establish a chain of evidence, ITS and VCS were used to maintain a record of all data of the study. Finally, outside researchers studying crowdsourcing software engineering were involved in the study and reporting of study results. In this case study, we make no causal inferences, so internal validity is not a concern.
External validity is the ability of a case studyâĂŹs findings to generalize to the broader population of interest [28]. A possible threat to external validity is that we only analyzed one project. Microtask programming is not inherently domain specific and would be relevant for other projects beyond developing web applications. Both of the key supporting tools (ITS and VCS) in the case study are widely used in software development projects. This supports the external validity of our study.
Reliability is the ability to repeat a study and observe similar results [28]. To reinforce our studyâĂŹs reliability, we defined and documented the microtask programming workflow and the approach for assigning microtasks to crowd workers. By using the same process and tools, other researchers or practitioners may replicate the case study in their own context.

DISCUSSION
In this paper, we investigated the potential of applying microtask programming in a company setting, reporting on a project undertaken at NTT to build a web application through microtask programming. Japanese IT vendors such as NTT face a serious IT talent shortage. Microtask programming may help to address this by increasing the fluidity of developer assignments within large organizations.
In adopting microtask programming, there were four key ways in which development work differed from standard practice: (1) a finer granularity of work, (2) less face to face communication, (3) associating contributions with artifacts, and (4) enabling developers primarily assigned to a different project to contribute in their free time. Contributions were smaller than typical, focusing on implementing or reviewing the behavior specified in an individual micro-specification. Rather than participate in the typical daily face to face or conference meetings to coordinate, crowd workers instead collaborated by reading tickets describing their work. Rather than work on tickets encompassing an issue spanning multiple source code artifacts, crowd workers instead worked on an individual artifact, reducing necessary project knowledge. And rather than work exclusively on a dedicated software project, seven crowd workers made contributions to a second project in their available free time. These differences led to a different style of work.
There were also differences in the nature of the review tasks. The focus of code reviews is traditionally on conventions, such as coding rules and format. In microtask programming, workers were also responsible to check whether the code satisfied its expected behavior. As a result, review tasks took longer than usual.
Adapting microtask programming did not require a change to the project life-cycle . Microtask programming was organized in the same four key project steps typically used by the organization (Fig. 3). The key difference was that microtasking added a new level of detail in the detailed design step, where the specifications were decomposed into micro-specifications.
The results demonstrate the potential for microtask programming to increase the fluidity of project assignments within an organization. The project met its objectives and successfully completed its system testing. Crowd workers assigned to other projects were able to contribute in their slack time and largely felt that the onboarding costs were reduced.
Prior to beginning the project, there were concerns about the quality of the code and the project's time to market. It was not clear how much the output of the crowd could be trusted. The project was built and tested to evaluate if the system satisfied its requirements. The system testing revealed only a few defects, which was considered quite successful. There were also concerns about the efficiency of microtask programming. It was not clear how much time would be needed to complete a project through microtask programming. However, in less than 2 months, the project completed 9 functions and 5 UI Components. This alleviated concerns that microtask programming might be excessively slow.
The project also revealed several challenges with applying microtask programming. The crowd workers reported that they found staying motivated to be harder. Crowd workers found the style of work used in microtasking to be unfamiliar and different. Motivation may also have been reduced because microtask programming was not typical practice and workers did not feel they needed to be as productive. Workers did not directly gain anything for their contributions, further reducing their motivation.
From the viewpoint of project management, it was difficult to monitor the status of the crowd workers because they were assigned to other projects. Only when they were not busy in their own project could they make contributions to the project. This made it hard to anticipate when and how often they would be able to contribute. As a result, estimating progress and handling risk in the project were very difficult.
While project participants perceived onboarding overhead to be reduced, there was still considerable overhead involved for crowd workers to get started and begin contributing. At the beginning of the project, workers were confused about the new concept and workflow. Workers also did not have their regular face to face meetings. It may be that after workers become more comfortable and familiar with microtask programming, some of this overhead may decrease. Moreover, while microtask programming often includes a pre-configured development environment, workers did not benefit from this in our study. Offering this might reduce onboarding overhead.
In this project, crowd workers manually selected microtasks from an issue tracking system. We are planning to develop techniques to generate these assignments, by considering crowd worker's main area of work and expertise and their current daily tasks.
In the project, the dedicated software engineer's workload increased during the detailed design step in which they created the micro-specifications. As there was no systematic method available for completing this work, the dedicated software engineer had to manually determine how to do this through trial and error, which increased their workload. This effort might be reduced through tool support, creating more detailed guidelines, or by adapting the workflow to find new ways to use the crowd.
Microtasking was only adopted for parts of the project which required less project knowledge to complete or that were not complex. Tasks that required more knowledge still involved a dedicated software engineer, as they required a level of knowledge that crowd workers did not have. This included 1) designing the system architecture, 2) designing and configuring the database, 3) completing tasks that involved screen transitions, 4) completing tasks which required infrastructure knowledge. In addition, the project's scope was limited to a development phase, rather than encompassing other phases such as maintenance. Better understanding how to employ microtasking in more complex programming tasks, when focusing on non-functional or crosscutting considerations such as performance, and in phases such as maintenance are essential questions for future work.

CONCLUSION
In this paper, we reported an industrial case study of the application of microtask programming to a web application development project. Work to implement a system was decomposed into a set of micro-specifications, which were then implemented and reviewed by the crowd. Crowd workers were primarily assigned to other projects and worked in their free time. A system with approximately 8,000 LOC in total was built, of which approximately 35% was implemented and reviewed by six crowd workers. Individual contributions made by crowd workers were small, averaging about 58 LOC.
The project results suggest the promise of microtask programming for making the assignment of developers to projects more fluid. Based on the results from this project, high-level management at NTT has realized the potential benefits. We plan to apply the approach to not only other development projects but also to system operation and maintenance projects. We are also planning to develop a tool for automating the integration of components implemented by crowd workers. We are considering potential approaches for managing progress and risk.