{'fraction_non_alphanumeric': 0.025986673500768838, 'fraction_numerical': 0.007585853408508457, 'mean_word_length': 5.166561314791403, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a Failure-Aware SDLC for Internet of Things

Internet of Things systems carry substantial engineering risks including catastrophic physical failures. To aid software engineers in developing reliable IoT systems, we conducted an experiment to evaluate the influence of learning treatments on design decisions. Specifically, we compared the influence of a set of design guidelines (current practice) and failure stories (proposed learning treatment) on developers' design rationale. We conducted an experiment with 21 computer engineering students using a questionnaire. We observed that both treatments helped subjects reason about criticality as a part of their design rationale. However, failure stories had a greater effect at enabling subjects to reason about safety as a part of their design rationale. We share our results illustrating the effects of a failure-aware design process and propose new research directions to enable a Failure-Aware Software Development Life Cycle for IoT development.


BACKGROUND
The Internet of Things (IoT), comprising smart devices interconnected with complex networks [31], has proliferated in modern societies [29]. In an IoT system, software interacts directly with the physical world, possibly autonomously, using distributed resources. A constellation of advances -in batteries, hardware, wireless networking, mobile computing, cloud services, and machine learning -has made widespread IoT systems feasible [14]. The worldwide IoT market is forecast to grow to an installed base of 42 billion devices in 2022 [16]. These trends have enabled IoT systems to become pervasive and increasingly interactive with the physical world where faults and defects are often safety-critical. Given the design complexities of IoT systems, their diverse characteristics enable diverse faults.
To ensure the reliability of IoT systems, it is necessary to follow reliable design practices. Recent IoT failures are a result of persistent single points of failure due to poor design practices: lack of redundancy [3,4,10,17], isolation [9,11,12,18,25], and authentication [2,9,11,23,32]. This pattern suggests that IoT developers may not be learning from reoccurring design failures. This is a sobering issue since IoT systems are often deployed in systems that are safety-critical, business-critical, and mission-critical [13].
Learning from engineering failures enables successful design [19]. In order to improve system design, lessons must be utilized from system failures [19,21]. Historically, lessons from failures have influenced system design in engineering disciplines such as civil, mechanical, and aeronautical engineering [19]. With the proliferation of IoT systems, software systems are increasingly safetycritical; thus, we promote the opportunity to borrow the practice of learning from system failures for the software engineering discipline. In the software engineering research literature, utilizing lessons from failures has been limited to postmortems practices for software project failures [6][7][8]30]. Similarly, we motivate the opportunity for postmortem practices for software system failures. Specifically, we seek to investigate the influence of system postmortems on design decisions and design.
To improve system design, we focus on design decisions and their rationales. For complex systems, design decisions can greatly impact the outcome of the designs [26]. Design decision rationales are used to understand the justifications, alternatives considered, and the trade-offs evaluated of design decisions [15]. Prior research to improve design decisions through rationale treatments have been limited to using general decision-making principles [28], reflective questions [20], and reminder cards with reflective questions [27]. Thus we inquire, could we improve software design decisions using design failures as a learning treatment?
We conducted an experiment to study the influence of failure stories on design decisions. We found that failure stories (proposed learning treatment) were just as effective as a set of design guidelines (current practice) at enabling subjects to reason about the criticality of design decisions. However, failure stories had a greater effect at enabling subjects to reason about the safety implications of their design decisions. Motivated by our results illustrating the effects of a failure-aware design process, we propose new research directions to facilitate a Failure-Aware Software Development Life Cycle in order to develop safe software for our increasingly smarter world.

METHODOLOGY 2.1 Problem Statement
Our goal is to examine the influence of failure stories on design decisions for IoT development:

Study Design
2.2.1 Compiling failure stories. We selected 2 recent IoT failure incidents reported in news articles to compile our failure stories. The first story narrated a fatal plane crash due to a lack of redundancy in a critical IoT system [10]. The second story described smart-car hacks that allowed access to driving functions due to improper segmentation of the driving systems [12]. We identified the context, the cause, and the impact of the failures from the articles. We presented this information as a short paragraph in Treatment 2's questionnaire. We also extracted design practices to mitigate these failures. These design practices were presented in Treatment 2's questionnaire as "lessons" -a common postmortem practice [30]. These design practices were also presented as a set of design guidelines in Treatment 1's questionnaire. The stories are included in §7.
To illustrate, we present one of the stories: Story 1: Recently, 2 airplanes crashed killing around 300 people. The cause of these crashes was identified as the interconnected stall protection system. This system was designed to stabilize the angle of an airplane from an unsafe upwards angle to a safe angle. This system consisted of one sensor to measure the angle, actuators to adjust the angle, a wireless network for communication, and a control system. The root cause of the failure was due to erroneous data from the sensor continuously triggering the system until the planes crashed. According to experts, a design redundancy of an additional sensor for this safety-critical system could have prevented the crashes. Lesson 1: Design redundancy for critical systems 2.2.2 Compiling design decisions. We created a hypothetical system design scenario to study the subjects' decision-making. The scenario depicted an IoT-enabled robotic e-commerce warehouse. We created 8 design decisions for subsystems in the warehouse. For each of the 2 failure stories, there were 4 design decisions that resembled the lesson outlined by the story, for a total of 8 design decisions. Also, 4 design decisions were for critical subsystems, whereas the other 4 design decisions were for non-critical subsystems. Each design decision consisted of two choices -a "correct" choice and an "incorrect" choice with respect to the criticality of the component and the lesson of a failure story. However, we note that these decisions are rarely binary, but rather ranked by criticality since any subsystem could be justified as critical. In order to account for this complexity, we also provided an open response field for the subjects to state their rationale for each of their decisions. In addition, we included a budget as a realistic constraint on the design decisions. The cost of each choice for a design decision correlated with the criticality of the decision: critical decisions necessitated a costlier choice, whereas the non-critical decisions did not.
To illustrate, we present one of the design decisions: Design Decision 1: Subsystem: Robot collision detection Subsystem Description: A system to detect and avoid an object in a robot's forward path. Design Question: The design team would like your help selecting the number of infrared sensors to use to detect objects in a robot's forward path. Please choose between the two options for the missing component: Option 1: 1 infrared sensor (Cost: 10,000 $) Option 2: 2 infrared sensors (Cost: 20,000 $) Please describe the reason for your decision.
The correct answer to this question is Option 2, since it is a safety-critical system resembling Story 1.

Study protocol.
The overview of the study protocol is outlined in Figure 1. This protocol was established after multiple rounds of pilot studies with 6 participants to adjust the treatments and design scenarios. A budget constraint was added to the experiment as a result of pilot study feedback. Subjects stated that without financial constraint, they simply chose the most precautionary option for each decision. This study protocol was approved by our institution's IRB. We recruited computer engineering students at our university through convenience sampling to respond to our questionnaire. Students were sampled at various stages of the undergraduate and graduate levels, inclusive of students with internship and full-time work experience. The questionnaire was distributed in a between-subjects design [1]. There were three groups:

Analysis
To determine whether the treatments influenced decisions, we studied the variance in the decisions between the three groups. Since our sample size was limited, we utilized the Kruskal-Wallis Rank Sum test. We tested with a null hypothesis that the decisions are the same between the two conditions ( = 0.5). Since design decisions are complex, we also studied the design decision rationales. With the qualitative data collected from the open response fields, we first performed open coding on a sample set of the data to establish a coding scheme [24]. This coding scheme was used to perform closed coding on all of the data [24]. Two authors conducted the qualitative analysis independently and the results were discussed to resolve conflicts.

RESULT AND DISCUSSION
We received a total of 21 responses to our questionnaire. Given our sample size, we frame our results as conjectures.
We examined the influence of failure stories on IoT design decisions. From an initial look, it appears that the decisions do not vary between the groups (as illustrated by Figure 2). This was confirmed by the Kruskal-Wallis Rank Sum test. Thus, we conclude that the quantitative analysis is not insightful due to limited insight gained from the binary choices and the limitation of the sample size for each group. To attain more insights, we conducted a qualitative analysis to code the open-ended responses about the rationale for the design decisions. As outlined in Table 1, we formed four codes: criticality, safety, cost, and performance. The distribution of responses labeled by the codes is illustrated in Figure 3. Subject used cost as a factor for decision Performance Subject used performance as a factor for decision We found that the failure stories influenced subjects' design rationales about criticality, safety, cost, and performance. We found that more responses from the Treatment 1 and the Treatment 2 groups reasoned about the criticality of the subsystems compared to the responses from the Control group. In addition, we observed a trend where more responses from the Control group reasoned about the safety implications of the decisions than the responses from the Treatment 1 group; however, a higher amount of responses from the Treatment 2 group reasoned about safety implications. Furthermore, we observed that the Control group was somewhat more concerned with the cost and performance implications of the decisions. Influence of failed design stories on design rationale concerning criticality. As illustrated in Figure 3, we found that both treatments helped subjects reason about the criticality of their design decisions. Almost twice the amount of responses from the Treatment 1 and Treatment 2 groups reasoned about criticality compared to responses from the Control group. We conjecture that this might be due to the mention of criticality in the treatments. It is also noteworthy that a larger amount of responses incorrectly judge criticality from Treatment 2 than from Treatment 1 with respect to our assessment. We conjecture that this might be due to an over-precaution for criticality, due to the catastrophic impacts of bad design detailed in the stories, as a result of a priming effect [22].
To illustrate the trend observed on design rationale concerning criticality in our study, we provide example responses. For example, a question tasked the subjects to make a decision about a crash alert system for robots with the option of only alerting the central warehouse management software or alerting the software as well as other nearby robots. We classified this subsystem as critical, necessitating redundancy with both alerts. However, a Control group subject was more concerned about cost, stating that it is: "Unlikely for multiple robots to...crash. Immediate human intervention can save $$. " A subject from Treatment 1, on the other hand, reasoned about the criticality of the system by identifying that "robot crashes are critical." Meanwhile, a subject from Treatment 2 reasoned about criticality as well as the impacts of the decision: "In case that a robot crashes, it would be important for other robots in its vicinity to know so they can avoid the crash. Otherwise, there could be more crashes... " Thus, we conjecture that failure stories are effective at helping developers reason about the criticality of design decisions.
Influence of failed design stories on design rationale concerning safety. As indicated in Figure 3, we found that the Treatment 2 more frequently reasoned about safety. It is also worth noting that the responses from the Control group considered safety more than the responses from Treatment 1. We conjecture that this is due to a lack of explicit constraints for the Control group subjects, which enabled them to freely brainstorm factors to guide their decisions, consistent with prior findings [5], and thus were more likely to consider safety implications.
To illustrate the trend observed on design rationale concerning safety in our study, we provide example responses. For example, a question tasked the subjects to make a decision about segmenting networks between personal and industrial devices, which could have safety implications under malicious circumstances. A Control group subject identified a malicious scenario -"Attack on one can save the other. . . " -implying that they recognize the safety measure to isolate the critical subsystem. Contrarily, the rationale of a subject from Treatment 1 was limited to the design guidelines provided and did not consider safety: "Separation of core devices, Better security. " Finally, a Treatment 2 subject identified the safety impacts of the decision: "If a worker clicks on a malicious link or gets a virus. . . on a personal device, this could be dangerous if industrial controls are on the same network. It is double the price, but safety should be a higher concern. " Use of anecdotal logic. An additional trend we observed in our results was that, across all groups, 8 subjects used anecdotal logic in their design rationales. For example, a subject in the Control group incorrectly cited our university's network as a basis to not segment personal and industrial networks: "Idk, I'm pretty sure all of the devices at [our university] share one or two connections so given that, it's probably good enough here as well. " In addition, a Treatment 1 subject incorrectly cited the concept of robot swarms as a justification for maintaining a single communication channel for all robots in the warehouse: "I believe this is what swarm means, to have all the robots be controlled under a single connection. " Meanwhile, a Treatment 2 subject cited the failure story we provided to choose redundancy for a safety-critical subsystem: "As we learned from the plane crash...design redundancy can help avoid terrible consequences. If the robots have only one sensor that becomes compromised, it could be hazardous for workers. " If design rationales use anecdotal logic, then failure stories could provide relevant anecdotes.

RESEARCH AGENDA
Our initial results illustrate the effects of a failure-aware design process for developing IoT systems. We envision precedent failures informing various development phases inclusive of requirements elicitation, design, implementation, validation, etc. We outline a research agenda with the first steps towards a Failure-Aware Software Development Life Cycle for IoT.
First, we propose an investigation into the effectiveness of guidelinebased practices at enabling developers reason about the safety implications of their design decisions. For example, a design experiment with a guideline-based treatment focused on safety-critical design decisions with complex constraints such as time, cost, and performance, to study participants' design process and rationale, would provide such insight. This investigation will refine efforts to propose safety-aware design practices.
Motivated by our difficulty studying design decisions at a binary level, we express a need to develop experiments to better measure and understand the rationale for design decisions influenced by failure-based learning treatments. For example, an experiment to study the influence of failure stories on failure mode and effects analysis (FMEA) would provide insight into whether learning treatments might improve developers' design rationale concerning critical design faults and ability to propose fixtures.
Lastly, we propose an investigation of processes for learning from design failures. If failure-based learning treatments are effective at instilling good design practices, then we need effective processes to use them in practice. First, we need to investigate the current processes used by organizations. For example, industries would provide insight into whether and how organizations currently document and learn from design failures. Next, we propose an investigation into knowledge transfer processes used to share the system postmortems across teams and organizations. Relative to knowledge transfer, we are also intrigued by the use of failure stories as an aid for training inexperienced developers with good design practices. For example, a design experiment for novice engineers, with the presence of failure stories as a control, to study the influence of the stories on their design decisions would provide such insight. These steps would enable us to study the effectiveness of such existing processes and propose a grounded failure-aware design process.

THREATS TO VALIDITY
Our sample size limits our findings to conjectures. Due to the nature of our experiment, our questionnaire surveyed for intentions rather than actions, and did not incorporate a full design cycle.

CONCLUSION
We investigated the influence of failure stories on design decisions. We found that failure stories were as effective as design guidelines at guiding developers reason about the criticality of design decisions. We found that design guidelines constrained developers reason about the safety implications of design decisions, whereas failure stories were conducive. Our observations about the effects of a failure-aware design process inspire and motivate new research directions in order to develop a Failure-Aware Software Development Life Cycle to help create safe software for our increasingly smarter world.