{'fraction_non_alphanumeric': 0.02448116833205227, 'fraction_numerical': 0.003920061491160646, 'mean_word_length': 5.198427822772749, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Complex estimation of status and operability of multilevel distributed systems

Methods of evaluation of condition and operability of complex distributed systems are considered in this paper. Author propose a new conceptual model of multi-stage complex performance evaluation based on multi-criteria function including various performance, reliability, stability characteristics indicating their contribution to general system performance. The approach used makes it possible to obtain an integrated assessment of system availability taking into account hardware and software reliability analysis as well as influence of various factors (hardware, software, operator’s errors) on overall system condition. Complex Population Notification and Warning System was considered as a subject of the research and was used to test the model developed.


I. INTRODUCTION (HEADING 1)
Modern technology makes it possible to increase the number of distributed systems with sophisticated structure which include various software and hardware modules and communication environment providing interaction between these modules. Information about each element's availability is a crucial indicator for assessment of operating state of such systems. Working order of elements determines availability and complete or partial functionality of the whole system.
With further development of information technology human's part in system efficiency and availability assessment decreases [1]. Earlier experts studied data collected from sensors and tested systems manually. Today software-based methods of efficiency and availability assessment of system modules predominate [2]. A module (or a subsystem) in this paper is a functionally complete element of a distributed system.
The most widespread method of availability assessment of system as a whole and each separate subsystem consists in defining a number of states: fault-free, faulty, available, not available and limit state on the basis of some approved criteria [3].
It should be noted that in addition to "Complete availability" and "Complete fault" states for the class of systems considered there are a number of intermediate states with various availability levels of its modules and corresponding performance indicators. All system states are to be clearly determined and logged which in real conditions represents a challenging task for experts in feature identification and criteria determination [4].
In order to determine criteria for the states of system's elements parameters for assessment are to be selected and corresponding coefficients are to be calculated. These parameters and criteria indicate current activity of certain element at any moment of time. Availability factor of each module of the system is to be calculated on an integrated basis taking into account a number of parameters which influence the possibility of continuing full-scale operation [5].
The study of the problem of determination of states and calculation of availability indicators of distributed systems began in 1950s since complex hardware-software devices appeared. In 1980s availability assessment of software component of complex equipment was performed on the basis of hardware reliability assessment methods developed earlier [6]. Research procedure was based on assumption that availability of software and hardware components are inseparably related and software component is just an element of hardware which can be a cause of failures [7]. There were no well-developed methods of reliability assessment of software component of the system. Methods and models of reliability theory [8] customized for hardware were used in the process of software efficiency and availability assessment. Mean time between failures (MTBF) [9] was the major indicator. of availability of its components regardless of importance or critical role of certain modules.
In 2000s simulation of system structure was performed regarding features of considered system and very often was represented in the form of an oriented graph. Functional degradation (failure of part of system's functions) in the process of system operation was taken into account resulting in model refactoring [13,14]. A number of availability criteria of system's elements was selected for each developed model on the basis of a set of uncorrelated parameters [15]. A number of assumptions in assessment methods used for system model optimization and simplification of calculation could distort (upwards) data on reliability and availability of elements [16]. These assumptions are given below: • method of combining elements with high and low failsafety in order to improve system reliability characteristics [16].
• mutual independence of separate elements and use of fault probability product theorem for evaluation of general system performance [17,18].
• representation of system model in the form of a graph with singly connected components and without multiple connectivity (cycles, disconnected nodes) [20].
Results of performed analysis indicate that there is a need to develop methods of integrated assessment of availability of multi-level distributed systems taking into account specific features of device operation, sophisticated integration of software and hardware making it possible to use vast theoretical and practical experience of system efficiency and reliability analysis accumulated in this field.

II. PROBLEM STATEMENT
In general, case real distributed systems can have the form of networks with sophisticated configuration [21]. Availability assessment for the majority of such systems cannot be performed on the basis of simple summation of availability values of their elements using multiplication theorem of probability as it can be done for systems with series-parallel connection of elements [17]. In the process of state calculation for such systems, logic and probabilistic calculus (LPC) can be used on the basis of independent registration of failure flow due to various factors influencing hardware and software [22]. One of the most promising approaches in system states identification consists in registration of various information indicators of availability of complex systems at different levels of abstraction and approval of corresponding criteria.
Integrated Population Notification and Warning System (IPNWS) is a typical example of complex-structure systems. Such systems represent a set of modules: reporting objects (computer centers of various levels integrated in one system via data-exchange network) and control objects (sources and recipients of information on threats), considered as finite nodes of the system. Modules consist of software and hardware complexes including various telecommunication equipment and safety devices.
Special elements called Data Transmission Facilities (DTF) are included in IPNWS in order to make use of various data transmission channels between certain module pairs possible as well as to improve communication channel utilization factor. DTF performance characteristics are determined by the quality of information delivery.
When the states of the systems are studied instead of the real set of objects and products we operate with a certain mathematical model which reflects the most significant properties of a real system [23].
Let us represent IPNWS as a disconnected oriented graph, then: distributed system can be considered as a set of modules represented by a set of nodes A, communication between nodes is provided by data transmission facilities represented by a set of edges B. Certain level of system hierarchy corresponds to each node of the graph, it indicates centralization degree and importance for the whole system Thus, formal representation of the system in the form of an oriented graph is as follows: The set of modules can be written as: where --th graph node, = 1 … .
Let us consider formalized model of a geographicallydistributed system in mathematical form.
A set of modules distributed in accordance with concentration levels can be written as: where total number of modules in the system belonging to levels [1. . ], where − total number of levels in the system,number of modules at level , = 1. .geographically-distributed system level number.
In general case each module is a set of objects (softwarehardware devices) of various types: where = 1. . , --th object type, = 1. . , number of objects of -th type in -th module. Let us denote a set of objects of various types by , then The number of -th type objects in the system considered = ∑ =1 , then the set of sizes of objects can be represented as = { 1 , 2 … }It should be noted that DTF may include several communication channels with individual technical characteristics.
Let us decompose the system down to low-level elements regardless of hierarchy level. Element in this case is an indivisible unit of the system performing specified functions. The term "Element" includes communication channels and hardware unit including system software functions.
A set of modules in decomposed model of a distributed system can be expressed as = { , }, and a set of DTF -= { ℎ, ℎ }; Thus the number of elements F of a distributed system can be represented as: Given model of the system makes it possible to combine connected nodes into a sub-graph of the basic graph which will remain a disconnected oriented graph (e.g. if we select one of the elements of level − 2as a root and all connected nodes of level will be selected as leaf elements). Such division of basic model into sub-graphs makes it possible to assess availability of a subsystemfrom objects (leaves) up to top-level structures. In case of an Integrated Population Notification and Warning System such division will make it possible to obtain availability assessment for interaction between reporting objects in any given district of the city, threat detection facilities on a potentially hazardous facility and regional dispatch center responsible for response coordination.

III. MATHEMATICAL MODEL OF AVAILABILITY ASSESSMENT
One of the most efficient methods of description of a distributed system in the process of availability assessment is to use simulation models and their mathematical abstraction, formulated on the basis of the term "logical element of the system" which performs certain functions instead of taking the set of software and hardware objects as basis. At the first stage of development these models make it possible to select and calculate efficiency and reliability parameters for each separate logical element. Then assessment can become more sophisticated as we replace abstract modules containing a number of elements with actual functional subsystems combining modules according to certain features and then to a general system model.
Obviously the quality of system availability assessment depends on degree of decomposition and how detailed the system model is. At the top level of decomposition the state of the system is determined by operation mode of the set of modules as well as functional condition of corresponding data transmission facilities. At the lower level system availability depends on availability of each element as well as technical parameters of each communication channel between modules.
System approach to hardware reliability assessment for distributed systems includes selection of a functional mathematical model describing possible operational modes of the system as well as system regime calculation methods including various parameters (MTBF, availability remaining safe operation life, recovery time (uptime), availability assessment, performance and quality factors) [7].
A set of states of system's element can be written as follows: where ( ) --th state of -th element at the moment of time , = 1. . ℎ, = 1. . + .
As failures are events which occur accidentally due to unfavorable development of some events that took place, the process of error occurrence and subsequent process of availability recovery can be regarded as a queuing system (QS). Mathematical model of system availability assessment from this point of view must generally be stochastic, i.e. reflecting operational conditions and characteristics of occurrence of failures of simulated object by means of random values distributed in accordance with corresponding distribution law.
Thus, in the process of availability assessment of QS elements we are to use a set of parameters reflecting equipment structure with sufficient accuracy. Various incoherent data of both objective (instrumental) and subjective control (results of technical inspections, researches in the framework of preventive maintenance, expert estimates) can be used to calculate these parameters. On the basis of this information a common database can be formed serving as a platform for integrated assessment of current state taking into account influence of various parameters and structural units on the whole system.
Each of the states is characterized by a set of parameter values, describing object's state and qualitative features for which quantitative assessments are inapplicable. As a rule the list of these parameters and features as well as tolerable limits of their values is determined in standards, technical documentation and design documentation.
Status in this paper is considered as a complex propertya number of parameters correspond to each status of the module. The following can be regarded as availability parameters for each element : -quantity of spare parts, tools and accessories (SPTA).
More detailed information on the impact of each parameter on availability of elements and list of operations necessary for calculation is given in corresponding sections.
The set of all states of system elements can be written as:

A. Mathematical Model Of Hardware Reliability Assessment
Operation of system's hardware and communication channels can be determined in accordance with [13] as cycle operation for intended purpose. Then regarding selected reliability indicators, the following can be considered as basic ones:

Functional status of element ( )
Checking results of functional failure tests is the most informative assessment of element's availability regarding requirements of design and engineering documentation. This check is to be performed regularly; current functional availability assessment can be obtained on the basis of results of this check.
The importance of functions performed by certain element for system's priority tasks depends on the place of this element in hierarchy and its importance regarding availability of the whole system. Weight factor for separate functions is to be introduced in order to assess general functional status of an element.
where l-type function availability indicator in -th element = { 0, 1, , --type function weight factor in -th element, the following condition is met ∑ = 1 =1 . Thus, "functional status" parameter can be represented as an analytical availability of element to perform its functions, values are expressed as percentage. It is obvious that a criterion of critical value is to be introduced; if a certain value is reached the element is no longer considered available: where ( ) − number of failed functions of -th type element in -th module, weight factor of -th element type in -th module, ∑ = 1 =1 2. Hardware operational availability factor . .
where availability factor (achieved availability in hardware) . .probability of the fact that hardware will be available at the moment of time ; ( ) -probability of fail-safe operation during the time period ; 3. Availability factor is determined as:

Mean time to recovery (uptime) (for all types of failures)
Faultless operation is the ability of hardware to perform its functions even in case of failure (complete or recoverable) of its elements.
Hardware fault tolerance and uptime characteristics are determined as follows: where --th system continuous operation period; system down time, caused by -th fault; Nnumber of faults. . . availability factor for a top-level element is calculated as a product of incorporated modules: where ,number of elements of one type.

Probability of faultless system operation ( )
For top-level element . ( ) reliability is calculated as a product of single and duplicated hardware units:

Quantity of SPTA
It needs to determine values of criteria such as: minimum total cost of spare parts; ensuring the requirements for uptime of system operation; space, that occupied by equipment from SPTA, demands a method of recruitment; other indicators of system reliability to evaluate the sufficient quantity and optimum composition of spare parts. Evaluation of SPTA is to resolve following system of equations: where = . . . + Δ is the availability factor with SPTA using, Δ -is the effect of spare parts use, is reflected in the increase/decrease of readiness of module/system, Δ

B. Mathematical Model of Software Reliability Assessment
Criteria and parameters of software availability assessment are different from the ones used for hardware. First of all, this can be explained by the difference between their operational characteristics. Based on some assumptions (about the size of computational resources and data storage capacities) we can say that software unlike hardware is not susceptible to wear. In general case software reliability increases with time as updates and new versions are released.
Step by step it approaches to a steady period as the error occurrence rate reduces. Software may contain errors (defects), however, if they do not come out, software can successfully perform its functions for a long period of time.
Software availability assessment includes the following terms: durability (program does not deteriorate due to external actions of user), faultlessness (errors do not influence on correctness of operation), correctness (either there are no errors, or their number decreases with time), correctness of input data (assumption on correct set of input data).
In order to obtain software availability assessment for the systems of considered class a simulation model based on Jelinski-Moranda and Schick-Wolverton software reliability models [24] which according to researches [25][26] are the most suitable for availability assessment of large-scale programs with a long debugging period. Assessment model assumes that existing errors can be corrected in the process of software operation as well as new ones can be made. This approach makes it possible to calculate analytically the number of defects left in software.
The basic assumption in the model is the fact that error detection rate is proportional to the number of defects left after − 1 time period, total debugging time up to the beginning of current period, average error finding time in current -th debugging time period , and makes it possible to estimate the probability of faultless system operation and software availability factor. It should be noted that the model we use would provide us with low (pessimistic) values of reliability assessment.
1. Probability of faultless operation is determined as: where count of registered failures, experiments count, proportionality factor; 0number of errors at the beginning of debugging stage;total number of time periods in which at least one error was detected. In our case, it is equal to total number of errors detected, as we made an assumption that only one error takes place in each time period.
In order to assess model parameters and 0 a set of equations is to be solved (iterative solution, then E0` is rounded and 0 is obtained): Mean time between errors detected: 3. Availability factor of software is determined as: . . = . .
mean time between failures for software, , .
mean time to recovery (uptime) after failure for software, . .
software availability factor (evaluated regarding subsystems) и determinates with term of steady-state operation (when → ∞).
--th period of continuous software operation; system down time, caused by -th error.
Achieved availability ratio for application software of the entire module can be calculated as achieved availability of functions, that module performed:

C. Model of Integrated Assessment of System Operability
The main difficulty in the assessment of the system's state consist in the formation of a single indicator of operability. Structuring the performance and reliability of a collection of different software and hardware systems that makes up the distributed systems is complicated by the dynamic model elements correlation to different modes of functioning of the system. It should be noted that the definition of clear criteria for the state of the entire system, including the concept of a complete failure, as is hard solve problem.
One possible approach is the following: determinate a set of elements that perform simple (standard) function. This set is based on graph , which present a functional structure of a distributed system IPNWS. The process of assessing performance is the recursion from the last (terminal equipment, performing the role of information) to the basic elements (control centers, address coordination messages and informing managers). In this approach, the entire distributed system is splited into its constituent elements, which are presented as consistent connection graph nodes in structural scheme of state's assessment model. The arcs of the graph that act as channels of communication between system modules, presented in the form of elements with only one function -data transfer. The relationship between the graph nodes and arcs will be presented in the form of a matrix effects (adjacency matrix) [ × ] = , , which reflects the priority of specific communication channel use.
The weighting factor enters to calculate the effect of a specific element in the overall state of the entire system. That factor equals to the weighted quantity of functions implemented by them, comparatively on all systems running functions. This approach takes into account the degree of influence of a single element failure level ( = 1. . ) on overall system reliability.
Research of operability will be carried out with the assumption that the stationary mode is set. In this mode, system can change the state, but the transition probability does not change: = lim →∞ ( ) Assuming the notion of equivalence complete system failure and total failure of all its elements, we can be formulate overall probability at time , expressing it, according to a consistent scheme of including all indicators: Since the elements of the system at different levels are almost typical (i.e. represented a monotonous structure in terms of set theory) and do not overlap with each other (including application software), we can get a final assessment of the availability coefficient as a weighted sum of the components, taking into account the importance of each element in the system: where meaningful factor of j-th module in system operability, besides ∑ = 1
Choose the meaningful factors should take into account that the failure of the entire system, from the point of view of the possibility of transmission through communication channels, will come after the failure of the DTF of the lower level of the system. The failure of one channel will be considered a partial failure (or refusal of one DTF), but will not be considered as failure of the entire system.
The resulting formula evaluation hardware and software elements of system state can analytically calculate the efficiency of the considered IPNWS system, consisting of a large number of subsystems with different characteristics of efficiency and reliability, recovered during the operation and associated multichannel data transfer subsystem.

CONCLUSION
An approach to integrated assessment of availability of multi-level distributed systems on the basis of a multi-criteria function taking into account various technical impact factors.
A method of integrated assessment of availability of software and hardware systems taking into account specific features of use of system and application software.
Availability parameters of elements obtained in the process of simulation of a distributed system were used for development of an integrated availability model of Integrated Population Notification and Warning Systems.
Distributed system model description is given in this paper using the example of IPNWS and developed availability assessment model for software and hardware components of the system consisting of devices, communication channels, system and application software.
The purpose of development of simulation model of a distributed system is to obtain qualitative availability assessment in order to ensure faultless operation by means of various measures improving efficiency (backup and sparing, preventive maintenance, modernization of equipment and communication channels etc.) The developed research procedure for availability assessment of a distributed system can be used for:  development of requirements to reliability, efficiency and availability of geographically-distributed systems  problem statement for development of structural diagrams of complex systems including reliability hardware and software;  development of mathematical models (logical, analytical and statistical) for simulation of failures and other emergency situations;  calculation of necessary reserves and SPTA to provide system operation in accordance with approved reliability and safety requirements;  use of obtained results for development and feasibility analysis of research, design, operational and other solutions.
This article accomplished in the context of government tasks by Russian Federation Ministry of Education and Science of the higher education institutions in terms of R&D on the theme "Research and development of new technologies, algorithms and software for navigation and telecommunication systems of new generation using the GLONASS navigation system and unmanned aircraft." № 4.1313.2014 -1201459777.