{'fraction_non_alphanumeric': 0.03667245705462266, 'fraction_numerical': 0.016673347883538966, 'mean_word_length': 4.5531371094072055, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modeling Impact of Human Errors on the Data Unavailability and Data Loss of Storage Systems

Data storage systems and their availability play a crucial role in contemporary datacenters. Despite using mechanisms such as automatic fail-over in datacenters, the role of human agents and consequently their destructive errors is inevitable. Due to very large number of disk drives used in exascale datacenters and their high failure rates, the disk subsystem in storage systems has become a major source of Data Unavailability (DU) and Data Loss (DL) initiated by human errors. In this paper, we investigate the effect of Incorrect Disk Replacement Service (IDRS) on the availability and reliability of data storage systems. To this end, we analyze the consequences of IDRS in a disk array, and conduct Monte Carlo simulations to evaluate DU and DL during mission time. The proposed modeling framework can cope with a) different storage array configurations and b) Data Object Survivability (DOS), representing the effect of system level redundancies such as remote backups and mirrors. In the proposed framework, the model parameters are obtained from industrial and scientific reports alongside field data which have been extracted from a datacenter operating with 70 storage racks. The results show that ignoring the impact of IDRS leads to unavailability underestimation by up to three orders of magnitude. Moreover, our study suggests that by considering the effect of human errors, the conventional beliefs about the dependability of different Redundant Array of Independent Disks (RAID) mechanisms should be revised. The results show that RAID1 can result in lower availability compared to RAID5 in the presence of human errors. The results also show that employing automatic fail-over policy (using hot spare disks) can reduce the drastic impacts of human errors by two orders of magnitude.


I. INTRODUCTION
The availability and reliability of Information systems is seriously affected by human errors [1], [2], [3], [4] where some field studies report human errors as the cause of 19% of system failures [5], [3]. Large datacenters with Exa-Byte (EB) storage capacity (by employing millions of disks drives) are expected to face at least a disk failure per hour. Mechanisms such as automatic fail-over try to reduce the role of human agent in service and maintenance tasks, however, in many cases the involvement of human is inevitable. Meanwhile, despite precautionary mechanisms such as using checklists and complying high standards for training the technicians, the human error probability (hep) is between 0.001 and 0.1 [6], [7], [8], [9]. These statistics translate into multiple human errors a day in an exascale datacenter. As a simple and frequent example of human error in datacenters, assume an array with one failed disk, and a human agent that is responsible for replacing the failed disk with the brand-new one. However, due to the lack of concentration, he or she wrongly removes the operating disk, rather than the failed one. This makes the whole array unavailable and can even lead to data loss if the wrongly replaced disk is thrown away [4].
The most vulnerable component in a Data Storage System (DSS) 1 is disk drive, where disk failures and Latent Sector Errors 2 (LSE) [14] cause the majority of Data Loss (DL) in data-centers. Investigating the effect of these two incidences on the reliability of disks drives and disk arrays have been the subject of several studies [15], [16], [14], [17], [18], [19], [20], [21], [22], [23]. Elerath and Pecht [16], [17] show that the conventional reliability estimation approach, Mean Time to Data Loss (MTTDL), can result in DL underestimation by orders of magnitude, as using MTTDL approach mandates assuming exponential distribution for both disk failure and fail-over rates, which is not realistic. In return, this study leverages the field data and shows that the rate of operational disk failure, LSE, disk fail-over, and Disk Scrubbing 3 can follow a three-parameter Weibull distribution. This work evaluates the reliability of Redundant Array of Independent Disks (RAID) using Monte Carlo simulations, but arguably takes the loss of one data stripe (by LSE) as a Double Disk Failure 4 (DDF) and finally counts the number of DDFs as a reliability metric, which results in data loss overestimation. Moreover, Elerath and Pecht [16], [17] just consider the single configuration of array having infinite cold-spares (mandating human assistance in disk fail-over), while ignoring the effect of human errors. Greenan et. al. [18] proposes NOrmalized Magnitude of Data Loss (NOMDL) metric, defined as the amount of data loss within mission time, normalized to the usable capacity of disk array, to cope with the limitations of DDF metric. Elerath and Schindler [15] extend the RAID5 models appeared in [16], [17], [20] to be applied to RAID6 arrays, by proposing a closed-form equation that uses a table of failure and repair parameters obtained by Monte Carlo simulations using Weibull distribution. One can conclude that the focus of all previous work is on DL in the disk array, ignoring the possibility of Data Unavailability (DU) caused by human errors.
Considering the effect of human errors alongside the knowledge provided by previous models and field studies, we can conclude that an accurate modeling of RAID dependability is very crucial to take into account several important criteria including a) a realistic distribution for failure and repair rates, b) the effect of LSEs and its differences with operational disk failures, c) the possibility of human errors in array service and maintenance, and d) evaluation of both reliability and availability within mission time while considering fair and meaningful metrics for reliability and availability. To the best of our knowledge, none of previous studies have addressed these concerns in a unified framework, while the effect of human errors is totally missed in the previous dependability models.
In this paper, we propose a dependability model for the disk arrays by considering the effect of disk failures, LSEs, and Incorrect Disk Replacement Service (IDRS) as a common sample of human errors 5 .
To this end, we analyze the possible combinations of operational 3 A task that removes LSEs by periodically reading the disk data and checking it with its parity, correcting the corrupted data using the parity and moving it to a new location, and mapping out the damaged sectors. 4 An event in which the whole data of RAID5 array is lost, due to the consecutive failure of two disks. 5 While the incorrect repair service can have many different roots and happen in many different conditions, in this work we focus on IDRS. disk failures, LSEs, and IDRS in a disk array. This analysis which is demonstrated by state diagrams, concludes that the combination of disk failure and IDRS can result in the unavailability of the whole array, while the combination of LSE and IRDS results in the unavailability of one or multiple data stripes, mandating a metric which is capable to project the magnitude of data unavailability as well as unavailability duration. We further define NOrmalized Magnitude of Data Unavailability (NOMDU), as the duration of data unavailability multiplied to the amount of unavailable data (in an arbitrary unit such as mega bytes) within mission time, normalized to the mission time and usable capacity of disk array. In our analysis, both disk subsystems with and without automatic disk fail-over are considered.
Using the proposed failure analysis, we conduct Monte Carlo simulations to evaluate NOMDU and NOMDL during mission time, by considering three-parameter Weibull distributions for the rate of operational disk failure, LSE, and IDRS, as well as the corresponding repair rates. Several important observations are obtained by the proposed model. First, it is shown that human errors can result in storage unavailability by order of magnitude (up to N OM DU = 10 −5 when human error probability is 0.1). The human error can also increase the probability of data loss, specially when the human error probability is more than 0.01 (human error probability of 0.1 can increase data loss by one order of magnitude). Second, the presence of human errors can contradict the conventional assumption about the dependability of RAID mechanisms, as the RAID configurations with greater level of redundancy suffer higher unavailability caused by human errors. Third, it is demonstrated that automatic disk fail-over, when on-line rebuilt is provided by using spare disks, can reduce the drastic impacts of human errors by orders of magnitude.
The model parameters are obtained from industrial and scientific reports alongside field data, which are extracted from the main datacenter of Sharif University of Technology (SUT) 6 [24], operating with 70 storage and computing racks (with more than 100PB storage capacity). This datacenter is equipped with SAB-SE [25] storage nodes 7 each of which supporting up to 72 disk drives, enabling the datacenter to support more than 27,000 disk drives.
Our contribution over the recent work [4] is as follows: • The proposed model is extended to consider the effect of a) LSEs for RAID5 arrays and b) RAID5 with spare disk. • Models in [4] assume a 100% survivable storage system 8 , while this work assumes the general case in which parts of data can be non-survivable. • For the first time, a novel metric, NOMDU, is proposed to assess the availability of data storage systems. • By considering the data object survivability as a model parameter, the proposed model reports availability and reliability in terms of NOMDU and NOMDL. • Monte Carlo simulation is used to assess NOMDU and NOMDL, rather than Markov models, while time-to-failure and time-to-repair is generated by considering Weibull distribution, obtained from field data and state-of-the-art reports. • Model presentation is revised to improve its understandability and applicability. The remainder of this paper is organized as follows. Section II represents background and related works. Section III elaborates the human error analysis in disk arrays using Monte-Carlo simulations. Section IV provides simulation results and the corresponding findings. Lastly, Section V concludes the paper.

A. Dependability Models of Data Storage Systems
Many research studies have tried to evaluate and improve the reliability of data storage systems (in particular, disk subsystem) by considering the failure cases that result in data loss [28], [29], [20], [17], [18], [21], [22], [23], [30], [31]. Metrics of data reliability used in the literature include a) MTTDL [28] which attempts to express the average time between data loss events, b) DDF [29], [20], [17], [21] which expresses the expected time between failures, c) percentage of RAID array failures within mission time [22], and d) Magnitude of Data Loss (MDL) [18] which is the amount of data (in bytes) that is expected to be lost within mission time. The other dependability parameter, data availability, expresses the fraction of time that data is accessible by customers [32]. Dependability of data storage systems can be significantly influenced by parameters such as the rate of component failures, the rate of recovery mechanisms, and the structure of redundancy mechanism used to tolerate component failures. A variety of redundancy and recovery techniques is employed in data storage systems to mitigate the consequences of component failures and decrease the probability of data unavailability and/or data loss. These mechanisms usually come with considerable performance, energy consumption, or cost overheads. Hence, designers manage to use system-level dependability models to measure the effectiveness of redundancy mechanisms applied to data storage systems reaching cost-effective redundancy techniques.

B. Human Error in Safety-Critical Applications
Human Reliability Assessment (HRA) [33] techniques are developed to attain a better understandability and quantification of human errors in a non-benign system. These techniques mainly focus on quantifying hep which is simply defined by Equation 1 [7]. By referring to hep values obtained by National Aeronautics and Space Administration (NASA), European Organization for the Safety of Air Navigation (EUROCONTROL), and United States Nuclear Regulatory Commission (NUREG), it can be concluded that the probability of human error is usually between 0.001 and 0.1 depending on the application and situation. However, for the most of safety-critical and enterprise applications, the reported hep is in the range of 0.001 and 0.01 [7], [8], [9], [6].
Finally, we can note studies inspecting and modeling the effect of human errors in enterprise systems such as nuclear power plants [34], [35], and studies trying to improve the maintenance and test quality of enterprise systems, in favor of maintenance cost and reliability/availability [36], [37].

C. Human Errors in Data Storage Systems
Human errors can threat the availability/reliability of DSSs in different components and situations, however, in this work we investigate the effect of IDRS which is one of the most prevalent types of human errors. Consider a RAID5 array with no spare disks, in which the failed disk should be replaced by the brand-new disk before starting the fail-over process. As shown in Fig. 1, the operator may wrongly replace the brand-new disk with one of the operating disks, rather than the failed one. This incidence, called IDRS, makes  two disks, the wrongly removed one and the failed one, inaccessible, resulting in the unavailability of the entire array. If the human error is detected, the array will be available by undoing the incorrect disk replacement. Otherwise, if the wrongly removed disk is damaged before the detection and recovery of human error, the entire array will be lost due to DDF.

D. LSE
Most studies in the field of data storage systems have focused on the failure analysis of disks, including operational failures and undetected errors [16], [38], [39], [40]. Operational failures occur due to faults in electronic and mechanical components such as heads and platters. These failures result in data destruction where disk head is unable to find the requested data. Using RAID configurations is a solution for alleviating the effect of operational failures on data storage systems [41].
In addition to operational failures, other types of errors such as bad head write and bit error can also damage disk sectors. Another cause of sector errors is environmental particles which may be placed between platter and head. In the case of a write operation, positioning the disk head within track gaps can corrupt several sectors. These types of errors, named LSEs, may lead to a data loss event upon a disk failure [39], [14]. Fig. 2 shows how an LSE can result in data loss in the case of a subsequent disk failure in the case of RAID5. Suppose that a sector of disk A is affected by LSE. If disk B fails before detection of recovery of LSE, the data of affected sector in disk A cannot be recovered, as RAID5 can just tolerate the failure of one disk. Error Correcting Code (ECC) [42], [43], disk scrubbing [44], and intra-disk redundancy [45], [46] can be used to reduce the probability of data corruption in the presence of LSEs. The LSE rate of a disk drive may vary in time, depending on several parameters such as disk age, disk model, and I/O characteristics [39]. We should note that some works on disk array reliability (such as Venkatesan and Iliadis [30]) ignore the effect of LSEs that results in misleading conclusions [18], [47], [16], [14], [19].

III. HUMAN ERROR ANALYSIS IN DISK ARRAY
In this section, we model the dependability of disk arrays using Monte Carlo simulations rather than conventional alternatives such as MTTDL and Markov models due to their extensive limitations and inaccuracies. Many previous studies have concluded that MTTDL is an obsolete metric for reporting Data Loss [2], [18], [48], [49]. The disk arrays have infinite failure states (due to having infinite combinations of sector failures, disk failures, and human errors) and modeling them with a closed-form MTTDL expression, and even a Markov chain is challenging and erroneous [50]. Furthermore, the disk failure rate is a function of time that makes using Markov chains erroneous [18], [47], while many previous works encourage using alternatives such as Monte Carlo simulations that have not this limitation [2], [18], [47], [16], [14], [19], [39]. In this section, we first introduce NOMDU metric for evaluating the availability of data storage systems. Afterwards, we propose our framework for evaluating the dependability of RAID5 and RAID6 arrays by considering disk failures, LSEs, and human errors. Finally, we discuss the dependability of general erasure codes and how our proposed framework can be employed for different code configurations.

A. NOrmalized Magnitude of Data Unavailability (NOMDU)
To access unavailability in a data storage systems, we need a metric to be applicable and comparable in different storage capacities, and contain the magnitude of unavailable data. The original availability/unavailability metric cannot be useful in the case of storage systems, for two reasons: Case A) Availability is a function of storage capacity, while a storage system with a larger capacity but the same architecture will have lower availability. Hence, different storage architectures with different capacities cannot be compared using DU metric. We take an example where two system engineers evaluate the availability of two storage subsystems using conventional availability metric. Assume Subsystem 1 (SS1) employs one RAID0(4 disks) array and Subsystem 2 (SS2) employs two RAID0(4 disks) arrays, while the arrays of both subsystems have exactly the same architecture and components. Assume A Disk stands for the availability of each disk, Aarray stands for the availability of one disk array, and ASS1 and ASS2 respectively stand for the availability of SS1 and SS2. Regarding RAID0 configuration, the array is unavailable when at least one of disks is unavailable. Moreover, in the conventional availability definition, when one of two arrays is unavailable, the whole system is considered unavailable (as unavailability metric does not deliver any information about the magnitude of data unavailability). In summary, conventional availability of SS1 and SS2 is as follows: ASS1 = A 4 , ASS2 = A 8 As the formulations of ASS1 and ASS2 show, the conventional availability is a function of system scale. Hence, two systems with exactly the same architecture but different scales have different availability values. Moreover, the availability does not change linearly with system capacity (system scale). Hence, the system engineers cannot obtain the availability of SS1, by simply normalizing the availability of SS2 to its capacity.
Case B) Unavailability metric cannot represent the magnitude of unavailable data. In many failure cases, only a part of storage data is unavailable, while the definition of storage availability/unavailability is limited to the availability of whole data (the storage is considered available when the whole data is available). We take an example to elaborate this shortcoming of availability metric when used in data storage systems. To this end, we evaluate the conventional availability of a data storage system employing a single HDD and a true remote backup (such as Cloud backup). Suppose two failure types of disk failure and LSE are possible in a HDD with the following definitions: a) disk failure: Time To Failure (TTF), Time To Recover (TTR), and b) LSE: Time Between LSE (TBLSE), Time To LSE Recover (TTLSER). The storage system is available when all its data is available, i.e., when no unavailability is caused by disk failure and LSE: The shortcoming of conventional availability metric, as shown in above formulation, is that both HDD failure and LSE have the same impact on system availability, while they cause totally different magnitude of data unavailability (the whole disk size versus a single sector size).
Here we define NOMDU, as the duration of data unavailability multiplied to the logical amount of unavailable data, normalized to the mission time and logical capacity of storage system, as shown in Equation 2. Hence, this metric can assess the availability of a storage architecture, regardless of its size and mission time.  Fig. 3 shows the proposed state diagram for assessing DU/DL in a RAID5 disk subsystem by considering the effect of disk failures and human errors. This model is evaluated using Monte Carlo simulations, as using Markov models can be erroneous due to its memoryless nature that prevents modeling non-exponential failure distributions such as Weibull [51], [16].
We have the same convention in naming the states in all state diagrams. The states in which the next failure results in DU/DL are named EXP and the states in which the next failure does not result in DU/DL are named OP. Upon the occurrence of the first disk failure, the system state will move from the operational (OP ) to the exposed state (EXP ). While being in the exposed state, a second disk failure will lead to DL event whereas a human error during disk replacement will lead to DU event. If the human agent successfully replaces the failed disk with the brand-new one, the array goes to the EXPr state, in which the disk fail-over can be started on the brand-new disk.
When the array is in the DU state, by recognizing the human error and removing it, the array switches to the EXPr state, in which the failed disk is correctly replaced by the brand-new one and the failover process can be started. However, if the wrongly replaced disk is crashed, a DDF happens and the array switches to the DL state. The time to crash the wrongly replaced disk is considered to have the distribution of d crash . Per DU incidence i, NOMDU is evaluated using Equation 3 and is added to the simulation statistics.
In this regard, Equation 2 is rephrased as follows: Where N OM DU is normalized magnitude of data unavailability within mission time, and N OM DUi is NOMDU imposed by DU incidence i.
Finally, when the array is in the DL state, the whole array data is lost due to DDF. In a non-survivable storage, that has no backup and mirror, in this case the array data is permanently lost. Hence, NOMDL is evaluated and added to the simulation statistics as shown in Equation 5: Where N OM DL nonsurvivable i is normalized magnitude of data loss imposed by non-survivable DL incidence i. In this regard, NOMDL within mission time is the aggregation of NOMDL imposed by individual DL incidence, as shown in Equation 6 In the case of DL in a survivable storage, that has at least one up-to-date backup or mirror, the array data can be recovered from the backup. In this case it takes Backup Recovery Time, dBR to recover the data of lost array over the remote backup, while Backup Recovery Time depends on the parameters such as the size of lost data, backup throughput, array throughput, and network bandwidth. The survived data is not lost in the user side, but is unavailable within recovery time. Hence, NOMDU imposed by survivable DL incidence i is evaluated as Equation 7.
In general, we can consider Data Object Survivability (DOS) [27], defined as the probability that a data object is survived during period of time (t). DOS(t) can be statistically interpreted as follows. Per DL incidence at the storage system level, a fraction of lost data, DOS(t), has a correct backup at mission time t, while the rest of data (1 − DOS(t) fraction of data) has no correct backup and is permanently lost. NOMDL metric is projecting the data that is permanently lost in the user side. Hence, in each DL incidence, NOMDL is a function of DL magnitude (size of lost data at the storage system level) and 1 − DOS(t), i.e., the fraction of data that has no correct backup, as shown in Equation 9. Moreover, in each DL incidence, DOS(t) fraction of data is not permanently lost, as it is recoverable from remote backups and mirrors. This fraction of data is just unavailable (DU in the user side) within recovery time. Hence, imposed NOMDU per DL incidence is a function of DL magnitude (the size of lost data at the storage system level), DOS(t), and DL recovery time (from backup), as shown in Equation 8.
Finally, total NOMDL and NOMDU per mission is evaluated respectively by the aggregation of NOMDL and NOMDU within mission time, as shown in Equation 6 and Equation 4, respectively.
2) RAID6 Analysis: The proposed model for RAID5 (Fig. 3) is extended to assess DU/DL of a RAID6 array in the presence of human errors and disk failures, as shown in Fig. 4. In the RAID6 configuration, two redundant disks are used to tolerate two consecutive disk failures. Hence, the data loss event happens in the case of Triple Disk Failure (TDF). In the normal operation of a RAID6 array (shown as OP + state in Fig. 4), one and two disk failures will bring the array to either OP1F and EXP2F states, respectively. OP1F stands for the state in which one disk is failed, but the array is still operational. In this state, another disk failure moves the array to the EXP2F state. In the OP1F state, a successful disk replacement moves the array to the OP1F R state, while an unsuccessful disk replacement moves the array to EXPF H state. The exposed state in this figure expresses that the array will continue servicing read/write requests. However, in the case of another disk failure before the performing the recovery process, a TDF happens that results in DL. Assessing NOMDU and NOMDL for each DU and DL incidence is similar to the case of RAID5 (Section III-B1). While the array is in the exposed state, a wrong disk replacement can make the array unavailable. Additionally, while the array is in OP +, a single disk failure followed by two consecutive wrong disk replacements can make the array unavailable. If the disk replacement is performed with no human error, the array goes back to either OP1F R and EXP2F R states, when it is in OP1F and EXP2F states respectively, while the occurrence of a human error in the disk replacement changes the array state to either EXPF H and DUF F H states. The array goes to DUF F H state when the combination of two disk failures and one human error happens and goes to DUF HH state when a disk failure is followed by two human errors.
EXP2F stands for the state in which two successive disk failures happen. In this state, another disk failure moves the array to DLT DF (triple disk failure). In the EXP2F state, two failed disks need to be replaced by the brand-new ones, while we assume that both disks are replaced simultaneously. A human error in the disk replacement process, regardless it happened on one or both disks, moves the array to the DUF F H state, while a successful disk replacement moves the array to the EXP2F R state. In the EXP2F R state, the array has two failed disks that are replaced with the brand-new ones, and the data of two failed disks should be recovered using other n − 2 operating disks. In this state, two disks can be recovered simultaneously, or be recovered one after another. Both approaches take a minimum time twice the minimum time of one disk recovery, while the latter approach has reliability benefits, as after the recovery of the first disk the array moves to OP1F R state and stays a shorter time in the EXP2F R state. In the first approach, the array remains in the EXP2F R until the recovery of both disks. Hence, we take the latter approach in our simulations, as shown in Fig. 4. Finally, DUF F H stands for the state in which user data is unavailable due to two disk failures and one human error in the array. In this state, by recovering from human error the array moves to the EXP2F R state. However, if the wrongly removed disk crashes, the array moves to the DLT DF state (triple disk failure).

C. Dependability of RAID5 Considering LSE
The model presented in Fig. 3 is extended to include LSE, as well as disk failures and IDRS, shown in Fig. 5. In the case a disk failure is followed by a human error, DU happens which can be assessed as described in Section III-B. DL is another possible incidence when two consecutive disk failures or the combination of LSE and disk failure (on two different disks) happen.
In the OP state, all disks are operating with no LSE. An operational disk failure switches the array state to the EXP state, while the time to transition from OP to EXP is a function of number of disks, n, and the distribution of time to disk failure, dDF . The array switches from OP state to EXPLSE state when one or more LSEs happen. The LSE can be recovered by data scrubbing, while the time to scrub, with the distribution of d Scrub depends on the storage maintenance policies and can have a minimum value which depends on the array throughput [16]. A disk failure after an LSE on a different disk results in the loss of data which is damaged by LSE (state DLF LSE ). Elerath and Pecht [16] take this incidence as DDF, while it has a different magnitude of data loss (and consequently a different recovery time in the case of survivable array) compared to DDF, resulting DL and DU overestimation. The survivable sectors can be recovered from backups while the distribution of time to recover sectors from backups, dSBR is a function of the number of lost sectors, backup throughput, array throughput, and network speed 9 . In the DLF LSE state, by assuming DOS(t) as the data survivability, the NOMDL and NOMDU imposed by DL incidence i is evaluated respectively by Equation 9 and Equation 8, while the Logical Size of Lost Data is equal to the Size of Sectors Affected by LSE.
A transition from EXPLSE state to EXP state is possible when the only disk affected by LSE fails. The time to failure of LSE affected disk can be different from operating disks, as it can have alternative causes such as Excessive Block Reallocation and can be measured using field data [16], while it has no explicit rate and is included in dDF [16]. Hence, the time to transition from EXPLSE to EXP also follows dDF . When one of n − 1 LSE-free disks fails, the combination of LSE and disk failure moves the array from EXPLSE to DLF LSE , where time to transition is a function of n − 1 and dDF . Note if more than one disk is affected by LSEs, a consequent failure of any disk moves the array to DLF LSE (hence, the time to transition is a function of n rather than n − 1) and there is no transition from EXPLSE to EXP 10 . Hence, both transitions from EXPLSE to EXP and DLF LSE is also a function of L, the number of disks affected by LSE.
In EXP and EXPr states, the occurrence of LSE before the completion of disk replacement or disk recovery can move the array to DLF LSE . However, Elerath and Pecht [16] express that this transition has a low probability and ignore it.
DLF F and DU states are the same as DL and DU states in Fig. 3, while the imposed NOMDL and NOMDU can be assessed by Equation 9 and Equation 8 in the case of DLF F , and in the case of DU , NOMDU can be assessed by Equation 3.

D. Dependability of RAID5 With Automatic Fail-over Considering LSE
The final model presented in this section belongs to RAID5 array with hot spare disk, in which the delayed disk replacement policy is employed. In this policy, the disk replacement is performed after the completion of automatic recovery (to the spare disk), when the single  point of failure is removed. Hence, this policy forbids the human error following disk failure which results in DU in the case of no spare (Section III-C). The state diagram for obtaining DU/DL using Monte Carlo simulations is appeared in Fig. 6.
Upon a disk failure, the array moves from OP to either EXP and DLF F upon the first and second disk failures, respectively. In the EXP state, the automatic recovery starts on the spare disk, by distribution dDF , while the service agent has to forbid changing the failed disk with brand-new one, before the completion of recovery. After recovery, the array moves to OPns, where the array is operational but no spare disk is available. In this state, failed disk replacement can be performed by the service agent. The successful disk replacement moves the array back to the OP state, while the human error moves the array to EXP he state. In the EXP he state, one operating disk is removed due to human error and the array is working with n − 1 operating disks. In this state, another disk failure and an LSE moves the array to either DUF H and DUHLSE states, respectively, while a successive human error in disk replacement moves the array to DUHH states. In the DUF H and DUHH states, the whole array is unavailable while in the DUHLSE state, only In OPns state, a disk failure and LSE moves the array to either EXPns and EXPLSEns states, respectively. In EXPns and EXPLSEns states the array has no spare, while successful replacement of the failed disk moves the array to either EXP and EXPLSE states, respectively. Unsuccessful disk replacement in EXPns and EXPLSEns states results in DU, moving the array to either DUF H and DUHLSE state, respectively.

E. Monte Carlo Simulation
In the MC simulations, the disk failure and LSE incidences are generated by assuming the desired failure distributions such as Weibull and exponential. After a disk failure occurrence, the recovery time is evaluated depending on the defined average recovery distribution. Fig. 7 illustrates an example of the MC simulation for a RAID5 (3 + 1) array. In case of DDF, i.e., two consecutive disk failures in the same array while the second failure is before the recovery of the first failure, a DL event happens (at 407 and 893 in Fig. 7), while the DL is recovered from backup if happened on survivable data (time 407) or is permanently lost if happened on non-survivable data (time 893).
In the case of single disk failure, the failed disk is replaced by a human agent. However, the occurrence of a human error in the disk replacement process, by the probability of hep, makes another working disk unavailable, resulting in the unavailability of the entire data array (time 326). The combination of LSE with disk failure and human error result in DL and DU, respectively. For example, at time 610 an LSE happen on disk2, while the failure of disk1 at time 648 results DL in the affected sectors, mandating the recovery of lost sectors from backup. Disk scrubbing is periodically performed on each disk and removes LSEs, while the exact time of removing each LSE is defined by considering a uniform distribution between start-time and end-time of scrubbing. For example, at time 500 an LSE happen on disk1 that is removed by scrubbing at time 530. N OM DU and N OM DL is evaluated for each failure incidence, and is aggregated within mission time.
The error of MC simulations is inversely proportional to the root square of the number of iterations as shown in Equation 10. The number of iterations can be adjusted by the target accuracy (error) and the given confidence level. Error of Monte Carlo simulation is obtained by Equation 10 [52].
In Equation 10, n is the number of iterations (in our case n = number of simulated arrays = 1000), δ is the standard deviation of the target values (NOMDU and NOMDL in our case), and Z α/2 is the t-student coefficient for a target confidence level [52].

F. Monte Carlo Transitions
The MC simulations can be applied to any failure and repair distribution, including exponential and Weibull. Elerath and Schindler [15] consider a two-parameter Weibull distribution for time to disk failures, LSEs, recovery of disk failures, and scrubbing, and show that this distribution better corroborates the field data, compared to the exponential distribution. This distribution assumes the probability density function as shown in Equation 11 where t is time, η is the characteristic life, γ is location parameter, and β is the shape parameter [53].
We use the base parameters obtained from field data by Elerath and Schindler [15], as shown in Table I. Note as Elerath and Schindler use two-parameter Weibull, we need to consider γ = 0 when applying Table I parameters to Equation 11.  For disk replacement and human error recovery, we also cannot assume a constant rate (exponential distribution), as by this assumption the probability of disk replacement and human error recovery in any time interval with the equal size is the same, which is not realistic. Hence, we also use Weibull distribution for disk replacement and human error recovery. The time to disk replacement, with the distribution of dDR, has no minimum value, as the human agent can change the failed disk immediately after its failure. Hence, we consider minimum time of 0 hours for the location parameter (γ = 0). We consider shape parameter (β) of 2 to have a right-skewed distribution, similar to the disk restore distribution. We consider the characteristic life of half an hour (η = 0.5), obtained from the storage service logs of Sharif University of Technology [24] datacenter, as a typical expected time for the failed disk replacement.
Time to recognize and recover the human error is denoted by dHER. As the human error can be recognized and recovered immediately, we consider minimum time of 0 hours for the location parameter (γ = 0). The shape parameter of 2 is considered to have a right-skewed distribution, and the characteristic life of one hour (η = 1) is considered regarding our storage service logs and interviews with datacenter technicians. Time to crash the wrongly replaced disk is generated by considering the shape parameter 1.4, and the characteristic life of one year (η = 8760), obtained by our storage service logs. The location parameter is 0 (γ = 0), as the wrongly replaced disk can be immediately thrown away. The Weibull parameters corresponding to disk replacement and human error is appeared in Table II.
Time to backup recovery in the case of DL in survivable storage, dBR, can also be characterized by a three-parameter Weibull distribution. In the case of DDF, the data of two failed disks is obtained from the backup. An alternative is to obtain the data of the first failed disk from the backup, afterwards, reconstruct the second failed disk using the XOR of n − 1 operating disks of the array. Assuming a network connection of 1Gbps between the storage and backup, and considering the array has eight 500GB SATA disks with 50M Bps speed, obtaining the data of failed disk from backup takes 10 hours. Considering the disks are connected to a 1.5Gbps data bus, it also takes 10.4 hours to reconstruct the failed disk using the XOR of n − 1 operating disks of the array [16]. Hence, a minimum time of 20 hours is required to recover a DDF from backup (γ = 20). We consider twice of the minimum recovery time as the characteristic life (η = 40), and consider the shape parameter of 2, to have a right skewed distribution. In the case of DL in disk sectors, caused by LSE, the distribution of recovery time, dSBR, depends on the size of lost sectors. As one sector typically has an small size of 4KB, the minimum backup recovery time depends on the minimum disk response time and the network delay, while we consider one millisecond for minimum sector recovery from the backup (γ = 2.7 × 10 −7 ), two millisecond for the characteristic life (η = 5.5 × 10 −7 ), and the shape parameter of 2 (β = 2) to have a right skewed distribution. The Weibull parameters corresponding to dBR and dSBR are appeared in Table III.

G. Applying Proposed Model to General Erasure Codes
In the previous subsection, we discussed the effect of human errors in RAID5 and RAID6 arrays and clarified how we use Monte Carlo simulations to obtain NOMDL and NOMDU for a specific array architecture by considering disk failures, LSEs, and human errors. However, both RAID5 and RAID6 schemes are in the category of Maximum Distance Separable (MDS) codes. Many alternatives of MDS codes are proposed in the recent years to cope with failure types observed in HDD and SSD arrays. Hence, it is of great importance that our proposed Monte Carlo framework cope with MDS codes in general case.
MDS codes, proposed in 70 th , offer the maximum possible hamming distance (hence, the maximum correction capability) while being separable, and have many alternatives such as Parity codes, Reed-Solomon codes [54], [55], or array codes, such as EVENODD [29], RDP [56], X-codes [57], B-codes [58], HVD codes [59], Liberation codes [60], STAIR codes [61], Sector-Disk Codes [62], and Partial-MDS codes [63]. RAID5 and RAID6 configurations are also in the MDS category by keeping respectively one and two redundant parities to respectively cope with one and two device failures in a disk array. In a RAID5 configuration, a row-wise code-word (Parity code) is stored in a redundant data chunk (or in general, data symbol). The redundant data alongside the actual data constitutes a data stripe. Blaum et al. [63] propose a Partial-MDS code that uses the conventional row-wise parity alongside a new concept of Global Parity to cope with the combination of both device failures and symbol failures. In general, we have a linear [mn, m(n − r) − s] code where m is the number of rows per stripe (code-word), n is the number devices in a stripe (including redundant devices), r is the number of redundant devices, and s is the number of global parities, as shown in Fig 8. In Partial-MDS codes (Fig. 8), the P (Parity) symbols are taken row-wise, while G (Global Parity) symbols are taken globally from all array members. Blaum et al. [63], Plank and Blaum [62], and Li and Lee [61] propose different approaches for encoding/decoding of Global parities by different complexities and I/O overhead. This code can cope with r device failures and s symbol failures in each codeword. We can put RAID5 in the category of Partial-MDS codes by considering r = 1 and s = 0. Similarly, we can put RAID6 in the category of Partial-MDS codes by considering r = 2 and s = 0. Briefly, we use the term P M DS(m, n, r, s) to refer to a Partial-MDS code with m rows, n devices, r row parities, and s global parities.
1) Overheads of General Erasure Codes: Depending on the number of row parities and global parities, PMDS codes come with different I/O overhead, computational complexity, and Effective Replication Factor (ERF 11 ), while the computational complexity and ERF is analyzed in the previous work [61], [62], [63]. In general, ERF of P M DS(m, n, r, s) is calculated by Equation 12. 11 ERF stands for the ratio of storage physical capacity over storage logical (useful) capacity.
2) Dependability Analysis of General Erasure Codes: In the general case, we can consider four failure types for a disk array: • Array Data Loss (ADL): This failure is similar to what we previously called DDF in the case of RAID5, and TDF in the case of RAID6, in which the whole array is lost. • Stripe Data Loss (SDL): is named after the failure case in which one or multiple stripes of disk array is lost. • Array Data Unavailability (ADU): is named after the failure case in which the whole array is unavailable due to human errors (IDRS). • Stripe Data Unavailability (SDU): is named after the failure case in which one or multiple stripes of disk array is unavailable due to human errors (IDRS). Consider employing P DM S(m, n, r, s) in a disk array as detailed in Table IV. By considering the definitions shown in Table V, the conditions of ADL, SDL, ADU , and SDU failures are summarized in Table VI. ADL happens in a very simple condition, when the number of failed devices (DF ) surpasses r (the number of redundant devices). SDL happens when ADL condition is not satisfied, but there exist at least one stripe in which the number of LSEs surpasses the maximum correctable LSEs. ADU happens when ADL condition is not satisfied, but the aggregation of failed devices (DF ) and unavailable devices by human error (HE) surpasses r. Note it is possible that both ADU and SDL conditions are satisfied in some cases, when the whole array is unavailable while some of array stripes is lost. Finally, SDU happens when ADU and ADL conditions are not satisfied and at least one stripe exists in which the number of LSEs does not surpass the maximum correctable LSEs, but its data is unavailable due to human error. Note it is possible that both SDU and SDL conditions are satisfied in some cases, when the array has at least one unavailable stripe and at least one lost stripe.
We conduct Monte Carlo simulations using the framework described in Section III-E and check the failure conditions appeared in Table VI to recognize ADL, SDL, ADU , and SDU failure cases. For each failure case, we record the size of lost data (in the case of ADL and SDL) or size of unavailable data and unavailability duration (in the case of ADU and SDU ), and finally calculate NOMDU and NOMDL at the end of simulation using Equation 3 through Equation 9.

A. Experimental Setup
Monte Carlo simulations are conducted for 1000 arrays of RAID5(7+1) and the Weibull parameters appeared in Section III-F (Table I, Table II, and Table III). Each experiment simulates 10 years (87600 hours) of mission time. The Monte Carlo simulator is implemented from scratch in C++ with respect to the logic represented in Section III-E. The results of this section are obtained for a non-survivable storage system (the definition of survivable storage systems and non-survivable storage systems is clarified in Section III-B), hence, the recovery from DL states is not possible (in Fig. 3, Fig. 4, Fig. 5, and Fig 6, transition from DLF LSE , DLF F , and DLT DF states to OP state, appeared in dashed-line,

B. Validating Monte Carlo Implementation
This is the first attempt of modeling the effect of human errors in data storage systems. Hence, to validate the Monte Carlo implementation, we compare the TDF within mission time obtained by our Monte Carlo implementation considering no human errors, with the Monte Carlo results obtained by Elerath and Schindler [15] for RAID6 array. In this comparison, we conduct the experiments for 1000 RAID6(14 + 2) array groups and consider all data loss events, including DF+LSE+LSE, DF+DF+LSE, and DF+DF+DF, as TDF (Elerath and Schindler [15] follow the same approach and consider all possible combinations of disk failure and LSE that result in data loss as TDF). In Fig. 9, our simulation results for 10-years mission time is drawn versus the results by Elerath and Schindler [15] for Disk A, Disk B, and Disk C models (considering the parameters appeared in Table I). As Fig. 9 shows, our Monte Carlo simulations report slightly higher TDF values compared to previous work (on average 11%). We also compare the DDF within mission time obtained by our Monte Carlo implementation considering no human errors, with the results obtained by Elerath and Pecht [17], [16] for RAID5 array. In this comparison, we conduct the experiments for 1000 RAID5(7+1) groups and consider both LSE+DF and DF+DF incidences as DDF (Elerath and Pecht [17], [16] follow the same approach and consider all possible combinations of disk failure and LSE that result in data loss as DDF). Hence, in the state diagram of Fig. 5, transition to both DLF LSE and DLF F states is considered as DDF incidence. Table VII compares the number of DDFs reported by Elerath and Pecht [17], [16] with the results of our simulation for the first year of mission time. In Fig. 10, our simulation results for 10-years mission time is drawn versus the results by Elerath and Pecht [17], [16]. As the figure shows, for η Scrub = 12, 48, and 168 hours, our Monte Carlo simulations report greater number of DDFs, while for η Scrub =  Fig. 11 reports NOMDU and NOMDL for RAID5 array, obtained by the model appeared in Fig. 5. The experiments are conducted for 1000 RAID5(7 + 1) arrays of Disk A, Disk B, and Disk C (Table I). We differentiate NOMDL caused by DDF and LSE+DF, respectively appeared in Fig. 11(b) and Fig. 11(c). Fig. 11(a) shows that by increasing hep by one order of magnitude, NOMDU almost increases by one order of magnitude. Meanwhile, increasing hep has less impact on NOMDL caused by DDF, and negligible impact on NOMDL caused by DF+LSE. By increasing hep from 0 to 0.001, the increase of both NOMDL caused by DDF and NOMDL caused by DF+LSE is negligible for all disk types. By increasing hep from 0 to 0.01 and 0.1, NOMDL caused by DF+LSE increases respectively by 1.0002x and 1.002x in arrays of disk A, 1.01x and 1.2x in arrays of disk B, and 1.07x and 1.8x in arrays of disk C. By increasing hep from 0 to 0.01 and 0.1, NOMDL caused by DDF increases respectively by 4.7x and 38x in arrays of disk A, 2x and 10x in arrays of disk B, and 5.3x and 44x in arrays of disk C. We can conclude that human error increases NOMDU by one order of magnitude, while it has no impact on NOMDL when hep is below 0.001, and this observation is almost regardless of disk type. However, when hep reaches 0.01 and beyond, it dramatically increases DL within mission time.

C. Effect of Human Error in Non-survivable Storage System
Another important observation is that NOMDL caused by LSE is five orders of magnitude smaller than NOMDL caused by DDF, while our simulation results show that LSE causes more than 90% of all DL incidences. We can explain this observation by different magnitudes of data loss in DDF and DF+LSE incidences. While DDF makes the whole array lost, DF+LSE results in data loss of one or multiple  (Table I)  stripes. This observation concludes that the approach proposed by Elerath and Pecht [16], [17] in taking both DDF and DF+LSE the same will result in serious DL overestimation.

D. Availability Comparison of RAID Configurations with Equivalent Usable Capacity
In this section we investigate whether human errors can change our conventional assumptions about the dependability of different RAID configurations. To this end, we compare NOMDL and NOMDU of RAID5(3 + 1), RAID5(7 + 1), and RAID1(1 + 1) configurations, considering equivalent usable (logical) capacity.
1) Applying the RAID5 dependability Models to RAID1: RAID1 system is implemented by mirroring the disk data in a redundant disk. Hence, it can be modeled as a one-failure tolerant system. Similar to RAID5, the data is lost in the case of DDF and disk failure combined with LSE, and the data is unavailable in the case of human error in disk failure recovery process. As such, the DU and DL is evaluated by the models presented in Section III-C and Section III-D, by considering n = 2. Fig. 12 compares the availability of three different RAID configurations with equivalent usable (logical) capacity, in the presence of human errors. The results are obtained for a storage by the usable capacity of 21000 disks, for three following configurations: a) 7000 RAID5(3 + 1) arrays, b) 3000 RAID5(7 + 1) arrays, and c) 21000 RAID1(1 + 1) arrays.
Another observation is that by increasing hep to 0.01 and beyond, NOMDL caused by DDF in RAID1(1 + 1) surpasses both RAID5(7+1) and RAID5(3+1). It means that in the environments with high probability of human errors, RAID1 is not only less available than RAID5, but also less reliable.

E. Effect of Automatic Disk Fail-over Policy
In this section, we report the effect of the automatic fail-over with hot-spare disk, when the service agent follows delayed disk replacement policy, as described in Section III-D. Fig. 13 compares Fig. 13: NOMDU and NOMDL caused by human errors for conventional RAID5 configuration and RAID5 with hot spare disk and delayed disk replacement policy. The experiments are conducted for 1000 RAID5(7 + 1) arrays of Disk A (Table I). We differentiate NOMDL caused by DDF and LSE+DF, respectively appeared in subfigures b and c.
the NOMDU and NOMDL of basic RAID5 array and RAID5 with hot-spare disk (for 1000 arrays of disk A). As the results show, using automatic fail-over policy can significantly moderate the effect of human errors. For example, assuming hep = 0.00001, automatic failover decreases NOMDU by five orders of magnitude as compared to the conventional RAID. Another observation is that automatic failover policy can also decrease NOMDL caused by human errors. The hep of 0.01 and 0.1 respectively increases NOMDL by 4.7x and 38x compared to the case of no human error, while by using automatic fail-over policy, hep of 0.01 and 0.1 increases NOMDL by 1.04x and 5.2x, respectively, as shown in Fig. 13(b).

F. Comparison with Previous Models and Field Data
In this section, we compare the results of our proposed model (considering human errors) for RAID5 array with the previous RAID5 reliability models, including conventional MTTDL model by Gibson [28], NOMDL by Greenan [18], and DDF by Elerath and Pecht [17], where none of them consider the effect of human error and subsequent DU/DL. Table VIII compares previous disk array reliability models with the proposed model for 1000 arrays of RAID5(7 + 1) and 10 years mission time for Disk A, Disk B, and Disk C. In this comparison, we assume a non-survivable storage system (clarified in Section III-B1) with no spare disk and typical value hep=0.001, while the rest of model parameters is appeared in Table I and Table II. As reported in Table VIII, only the proposed model considers the effect of human errors and corresponding DU. As an example, the proposed model reports that for RAID5(7 + 1) arrays of Disk A, VIII: Comparison of previous disk array reliability models with the proposed model for 1000 arrays of RAID5(7 + 1) and 10 years mission time. We assume typical value hep = 0.001 and no spare disk in this comparison, while the rest of model parameters is appeared in Table I, Table II, and Table III Finally, to further show the shortcoming of previous works in neglecting the effect of human errors, we compare the proposed model results with previous work and field data from enterpriselevel storage products of a leading storage system manufacturer and storage service provider (here we call this company by CorpX), as shown in Table IX. Field statistics on the failures of four enterpriselevel storage series of this company roughly report that 15% of all data loss and data unavailability is caused by human errors.
As this comparison shows, the DL prediction of Greenan [18] and Elerath [17], [16] method is lower than the proposed model, as Greenan and Elerath predict no DL caused by human errors (they just consider DL caused by device failure and LSE). Consequently, total DL reported by the proposed model is 13% greater than Elerath [17], [16] and Greenan [18]. The more significant shortcoming of previous works, however, is ignoring the effect of data unavailability caused by human errors. The CorpX field data reports that 15% of total storage unavailability is caused by human errors, while the previous models do not consider the human error impact by any means. Comparing the proposed model results with the field data shows that total DL reported by the proposed model is in the same order with the field data when we choose hep = 0.001 and η crash = 10h. We are satisfied with this result, as CorpX also reports the average human error probability in the same range (0.02% to 0.1%). These results are reported for RAID5(7 + 1) configuration while the field data for other erasure codes are not available. The field statistics of Data Loss breakdown, obtained by DeepSpar [65] (a data recovery firm) from a survey of 50 data recovery firms shows that 12% of data loss in disk subsystems is caused by human errors [65]. This statistics is also in the same order with the proposed model results. The proposed model shows 12.8% of DL is caused by human errors when considering hep=0.001. We can conclude that by considering hep = 0.001, the proposed model results are accurate estimate to the field reports. This observation corroborates our previous hep evaluation based on human error statistics from Sharif data-center and related reports on human errors in the field.

G. Comparison of Monte Carlo Simulation and Markov Model
In this section, we compare the results obtained from Markov model with Monte Carlo simulations. In this regard, Markov model of RAID5 array (assuming no spare disk and not survivable data, i.e., DOS(t) = 0) is solved by algebraic approach and then NOMDU and NOMDL are obtained. The Markov model state diagram is same as Monte Carlo simulation state diagram (shown in Fig. 5) by considering exponential failure distribution (rather than Weibull distribution used in Monte Carlo simulations), with transition rates appeared in Fig. 14. The model parameters are appeared in Table I,  Table II, and Table III for Weibull distribution. To have a fair comparison between Monte Carlo simulation and Markov models, we justify MTTF/MTTR in exponential distribution to result in the same number of failures as Weibull distribution does in a 10-years mission time. In this regard, both Weibull and exponential distributions should have the same Cumulative Distribution Function (CDF) in ten years, as shows in Equation 13.
Where t is time, η is characteristic life, β is shape parameter, and M T T F is Mean Time to Failure. M T T R is obtained by the same equation. Then we set t to 10 years (87600 hours) and calculate M T T F and M T T R of exponential distribution. As such, both Weibull and exponential distributions generate the same number of failure/repair incidences (disk failure, LSE, disk repair, and scrubbing) within 10 years mission time. Fig. 15 shows Markov model results and the error of Markov model with respect to Monte Carlo simulation results. The error bar (in red color) and error percentage (appeared beside each bar) is also included in this figure. As the figure shows, Markov results have up to 97% error (in NOMDL DF+LSE

H. Model Results For Global Erasure Codes
In this section, we examine the dependability of general erasure codes presented in Section III-G. In addition to RAID5 (P M DS(m, n, 1, 0)) and RAID6 (P M DS(m, n, 2, 0)), here we examine P M DS(m, n, 1, 1), P M DS(m, n, 1, 2), and P M DS(m, n, 2, 2), by considering the effect of disk failures, LSEs, and human errors. We choose P M DS(m, n, 1, 1) and P M DS(m, n, 1, 2) that have a slightly greater ERF than RAID5, but considerably lower ERF than RAID6. Both P M DS(m, n, 1, 1) and P M DS(m, n, 1, 2) can cope with one device failure and respectively one and two symbol failures (due to respectively having one and two Global parities). P M DS(m, n, 2, 2) has a ERF greater than both RAID5 and RAID6, while it can cope with two device failures alongside two symbol failures per code-word.
Using the framework described in Section III-E, we conduct Monte Carlo simulations and check the failure conditions appeared in Table VI to recognize ADL, SDL, ADU , and SDU failure cases and finally calculate NOMDU and NOMDL. In summary, by considering ADL, SDL, ADU, and SDU statistics, we obtain NOMDU and NOMDL as shown in Fig. 16. One important observation in the NOMDU and NOMDL results of different erasure codes is that the codes with the same number of row parities have almost the same NOMDL and NOMDU value. We can justify this observation by the fact that the magnitude of data unavailability and magnitude of data loss caused by device failures is significantly greater than stripe failures. In specific, per ADL event, the magnitude of data loss is 8TB (assuming 1TB disks and array size of 8), versus 128KB per SDL event (hence, the magnitude of ADL is 62,500,000 times greater than SDL). This fact results in the superiority of the effect of ADL and ADU events in the final NOMDU and NOMDL values. For example, NOMDL of RAID6 and P M DS(2, 2) is very similar (4.05249887 × 10 −5 and 4.0524983 × 10 −5 , respectively), as both arrays perform the same in ADU and ADL, but different in SDU and SDL, due to having the same number of row parities and different number of global parities. We can also observe that in all erasure codes, human error increases both NOMDL and NOMDU by almost one order of magnitude that corroborates our previous observations on RAID5.

V. CONCLUSION AND FUTURE WORKS
In this paper, we investigated the effect of incorrect disk replacement service on the data unavailability and data loss of disk subsystem by using Monte Carlo simulations. We also proposed NOMDU, as the duration of data unavailability multiplied to the logical amount of unavailable data, normalized to the mission time and logical capacity of storage system, as a more useful availability metric for storage systems. By taking the effect of incorrect disk replacement service into account, it is shown that human errors can cause the unavailability of storage array by order of magnitude. The human error can also increase the probability of data loss, specially when the human error probability is greater than 0.01. It is also shown that in case the human error probability is high (0.01 and beyond), the conventional dependability ranking of RAID configurations is contradicted. Lastly, the model results show that automatic fail-over can significantly decrease the data unavailability and data loss, caused by human errors, by orders of magnitude. Such information can be employed by both designers and system administrators to increase the system dependability. r < DF (14) B. SDL Condition SDL happens when ADL condition is not satisfied, but there exist at least one stripe where the number of LSEs surpasses the maximum correctable LSEs. Stripe v has the following number of LSEs: h, is the number of operational devices that are behaved as failed device and all of their LSEs (regardless of the number of LSEs in that device) are corrected using row parities. To attain the maximum possible correction capability, we select h devices that have the maximum number of LSEs. Hence, the maximum correctable LSEs using row parities is as follows: Finally, SDL happens when the following condition is satisfied:

C. ADU Condition
ADU happens when ADL condition is not satisfied, but the aggregation of failed devices (DF ) and unavailable devices by human error (HE) surpasses r:

D. SDU Condition
SDU happens when ADU and ADL conditions are not satisfied and at least one stripe exists where the number of LSEs does not surpass the maximum correctable LSEs, but its data is unavailable due to human error. For satisfying SDU condition, at least one human error is happened and ADU and ADL conditions are unsatisfied: (0 < HE) ∧ (DF + HE ≤ r) (20) Moreover, the stripe v has no lost sectors under the following condition (as discussed in the case of SDL): Finally, stripe v has unavailable sectors under the condition that the number of LSEs in the available devices does not surpass the maximum LSEs obtainable with the available devices. The number of LSEs in the available devices is as follows: Maximum LSEs obtainable with available devices is the aggregation of LSEs obtainable with global parities and LSEs obtainable with row parities. The number of LSEs obtainable by global parity is equal to s (the number of global parities). However, the number of LSEs obtainable by row parities is a function of the number of failed devices (DF ), number of unavailable devices due to human error (HE), and the distribution of LSEs in the stripe. Using P M DS(m, n, r, s), in each stripe we can behave h number of operational devices as unavailable device and obtain all their LSEs using row parities (regardless of the number of LSEs in that device), where: To obtain the maximum possible LSEs, we select h operational devices that have the maximum number of LSEs. Hence, the maximum obtainable LSEs using row parities is as follows: And the maximum obtainable LSEs in stripe v is the aggregation of s and above value. Hence, stripe v has unavailable sectors under the following condition: All in all, SDU happens when the following condition is satisfied:  Fig. 17 and Fig. 18, we respectively draw the cumulative number of ADL and SDL incidences for RAID5(7 + 1), RAID6(7 + 2), P M DS(m, 8, 1, 1), P M DS(m, 8, 1, 2), and P M DS(m, 9, 2, 2), respectively denoted as RAID5, RAID6, P M DS(1, 1), P M DS(1, 2), and P M DS(2, 2) in the charts. To have a fair comparison, the erasure codes are considered to have almost equal usable capacity of seven drives (note PMDS codes have a usable capacity slightly lower than 7, due to the overhead of Global Parities). The simulation parameters are appeared in Table I,  Table II, and Table III. As the number of ADL incidences depends on the number of row parity devices, the erasure codes with the same number of row parities result in the same number of ADL in each fault injection experiment. Hence, we concatenate the ADL of RAID5, P M DS(1, 1), and P M DS (1,2), and also concatenate ADL of RAID6 and P M DS (2,2) in Fig. 17.
The first set of results is obtained for 10,000 disk arrays working for 10 years (87600 hours) considering the real capacity of each disk (Disk A: 1TB, Disk B: 1TB, Disk C: 288GB), shown in Fig. 17(a) and Fig. 18(a). As we see in the first set of results, the failure cases such as multiple LSEs in the same stripe and triple device failure are so rare. Hence, in practice we see no difference between the results of RAID6, P M DS(1, 1), P M DS(1, 2), and P M DS (2,2). To increase the chance of such failure cases, we decrease the disk sizes by the factor of 64X (we call it small disk size). Decreasing the disk size also decreases the simulation time, which makes simulating larger number of disk arrays practical.   Fig. 18(b) respectively show the ADL and SDL for 1,000,000 disk arrays with small size. In the results obtained by small disk size, we can apparently observe the superiority of P M DS(1, 2) and P M DS(2, 2) in preventing SDL events (zero number of SDL in our experiments), due to employing two global parities that cope with two sector failures per stripe. The results also show that P M DS(1, 1) outperforms RAID6 in handling sector failures. For example in the case of disk A, P M DS(1, 1) encounters 79 SDL events versus 193 SDL events observed in RAID6 array. In the case of array data loss, however, the number of ADL events is a function of employed row parities (employed redundant disks). Hence, we can see that RAID6 and P M DS(2, 2) outperform the rest of codes by almost one order of magnitude, due to employing two redundant devices rather than one redundant device in RAID5, P M DS(1, 1), and P M DS (1,2). For example in the case of disk A, RAID6 and P M DS(2, 2) encounter 46 ADL events versus 818 ADL events observed in the case of RAID5, P M DS(1, 1), and P M DS (1, 2).
Finally, the results of Fig. 17(c) and Fig. 18(c) are obtained by decreasing the disk sizes by the factor of 16384 (we call it ultra-small disk size) for 100,000,000 disk arrays. In the results obtained by ultrasmall disks, we can further observe the superiority of P M DS(2, 2) over P M DS (1,2) in handling sector failures. For example in the case of disk A, we observed 10 SDL events in P M DS(2, 2) versus 19 SDL events in P M DS(1, 2), as shown in Fig. 18(c).