{'fraction_non_alphanumeric': 0.03724315068493151, 'fraction_numerical': 0.008722174657534247, 'mean_word_length': 5.502783576896312, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fundamentals and contributions to validation of constructive computational methodologies: a focused survey

Computational mechanisms (complex structures of functionally connected algorithms) are developed very frequently by both the academia and the industry for specific applications, but the transferability to other applications is seldom addressed and systematically tested. This raises the need for application validity of constructive computational methodologies (CCMs). First, this background research paper clarifies the fundamental concepts related to application validation. Applicability validation focusses on the indicators of appropriateness with regards to a particular purpose. Then, it surveys the various approaches of application validation based on the publications available in contemporary literature. Its main finding is that applicability validation of CCMs seems to be a stepchild of academic research. The same applies to the industrial exploration of the applicability of CCMs tailored to a narrow family of applications to a wider range of applications.


Introduction
To support various knowledge-intensive activities during the life cycle of artifacts, constructive computational methodologies (CCMs) are developed [1]. The activities may concern requirement engineering, ideation, functional conceptualization, system architecting, detail design, simulation, optimization, implementation, operationalization, etc. The artifacts can be physical products (customer durables, electromechanical devices, working prototypes, etc.), software products (control software, middleware, application software, etc.), cyberware products (ontologies, knowledge warehouses, image repositories, etc.), and service products (mail service, transportation service, bank service, etc.). CCMs typically include five major constituents: (i) one or more underpinning theories, which provide a conceptual basis and a logical framework, (ii) a procedural scenario, which arranges the activities identified and supported by the methodology, (iii) a collection of methods, by means of which the activities can be executed in a structured and effective manner, (iv) a selection of instruments, which provide technical aids for the execution of activities, and (v) a specification of the constraints and goodness criteria for applicability and utilization of the concerned methodology ( Figure 1).
The above-mentioned testing/proving of a computational methodology boils down to a sequence of justification, validation, and consolidation (J-V-C) activities. These three activities include distinct sets of confirmative actions, but complement each other from an epistemological and a methodological point of view. They focus respectively on the assessment of (i) logical properness (trustworthiness), (ii) argumentative appropriateness (adequacy), and (iii) outward relationships (agreement). The given order of consideration in the confirmative stage of research and/or development is implied by epistemological logical arguments. Though justification and validation are interconnected, they have distinct roles in demonstrating properness and appropriateness. Therefore, this paper focuses on validation only.

Disambiguation of the concept of validation
The word 'validation' is frequently used in publications intuitively or ambiguously. Authors often replace the term 'validation' by synonyms such as certification, attestation, authentication, or confirmation. In the tradition of scientific inquiry, validation is a multi-faceted activity focusing on confirmation of knowledge. As Barlas, Y. and Carpenter, S. stated, while verification refers to internal consistency, validation refers to the appropriateness of knowledge claims [2]. In the process of establishing scientific theories, validation is done to test and prove their appropriateness and utility for a purpose in some (application) context. It seems that theoretical and pragmatic interpretations of these two notions exist concurrently. Pragmatic interpretation is predominant in the engineering and design practice, where verification simply means testing against requirements, while validation is checking the fulfillment of the expected functionality [3].
In simple words, the main question of validation is whether a new body of knowledge (theory, framework, methodology, etc.) does what it is supposed to do? This may not be achievable because of biases or lack of adequacy. Thus, validation should focus on the critical factors (possible sources and forms of biases) that influence the conduct as well as the status of the outcome (findings) of research in contexts (various real-life situations). The major concepts and the related representative approaches are graphically visualized in Figure 2. Normally, internal validation and external validation are distinguished. Internal validation aims at exploring and evaluating biases, which cause a theory not doing what it is supposed to do. The conduct of internal validation can be characterized by its temporal dimension and periodicity, respectively. External validation checks issues related to generalizability and reusability and provides information for consolidation of theory.
In general, consolidation evaluates how strongly the conducted research and its outcome relate to other investigations of a phenomenon and the state of knowing. Consolidation also has internal and external aspects. The former is related to the investigation of the theory in the subsequent phases of the conducted research, whereas the latter concerns the relationship of the new theory with the current epistemology. Consolidation also informs about what may be expected to occur in other research contexts. Thus, consolidation can be done from the perspective of a conducted research project, but it may also be directed towards more general disciplinary issues and contexts (i.e., how the theory fits into a broader picture?). Due to this dual objective, it may involve both specialization and generalization of the validated knowledge.
Many aspects of justification and validation of CCMs have been addressed and covered with knowledge in the related literature. Applicability validation is among them, but not with specific attention to constructive computational methodologies. This is an important fact since the input information for methodology development is typically obtained by considering only a limited number of cases, while the broadest possible range of applications is expected when the methodology has been developed. This is known as the methodology incongruity phenomenon in the literature. Though important from the perspective of industrial use of methodologies, application validation is challenging due to the shortage of efficient testing approaches. Therefore, an extensive literature study IOP Conf. Series: Materials Science and Engineering 573 (2019) 012108 IOP Publishing doi:10.1088/1757-899X/573/1/012108 3 and web search have been conducted to see the available approaches and to get insights into the opportunities, constraints and major issues.

Developments concerning systematic and rigorous validation
Five subject areas for validation have been identified, which are validation of: (i) data, information, and knowledge, (ii) concepts, theories, and models, (iii) objects, structures, and systems, (iv) actions, processes, and services, and (v) methods, methodologies, and tools. This served as the reasoning model for systematic exploration.

Validation of data, information, and knowledge
Input data and output data validations are routinely done in computer science. The former ensures that data inserted into an application satisfy the defined formats and other input criteria, whereas the latter focuses on the prognostic output from numerical models or computational algorithms. The applied methods are jointly determined by the type of data and the purpose of validation. For real-time validation of high-frequency data, Horn, W. et al. considered: (i) time-point-based methods, (ii) timeinterval-based methods, (iii) trend-based methods, and (iv) time-independent methods [4]. Sensor data/information validation is a typical task in the context of CPSs, which includes two operations: (i) basic validation (i.e. detecting the presence of a fault), and (ii) separation (isolation of faulty elements). Ibarguengoytia, P.H. et al. developed an anytime algorithm for this purpose [5]. Information validation is a critical semantic evaluation of data with respect to the degree of agreement among sources. Mengshoel, O.J. made an inventory of the principles and practice of knowledge validation [6]. Donald, J.G. discussed disciplinary differences in knowledge validation [7]. On the computational side, Garcia, A.C. and Vivacqua, A.S. argued that knowledge validation is still a challenge at constructing knowledge-based systems [8]. As a specific issue of knowledge validation, Centobelli, P. et al. proposed a 3D fuzzy logic methodology to evaluate the level of misalignment between an enterprise's knowledge and the knowledge management systems it adopts [9]. IOP Conf. Series: Materials Science and Engineering 573 (2019) 012108 IOP Publishing doi:10.1088/1757-899X/573/1/012108 4

Validation of concepts, theories, models
Model validation is about how closely a model used for exploration, prediction and/or prescription mirrors the represented reality (fidelity), and consequently, how well the information obtained by processing the model complies with reality (reliability). As McCarl, B.A., and Nelson, A.G. argued, fidelity and reliability imply the need for a priori tests and post priori tests [10]. This resonates with the opinion of Pace, D.K., who argued that validation has (i) conceptual validation aspect (when the anticipated fidelity of a conceptual simulation model is assessed), and results validation aspect (when the results of simulation by the implemented model are compared with an appropriate reference) [11]. If both end with positive confirmation, then the model supports the intended use. Balci, O. emphasized that verification, validation, testing, accreditation, certification and credibility assessment activities of models and simulations primarily deal with the measurement and assessment of their accuracy [12]. Thus, model validation is substantiating that a model, within its domain of applicability, performs with satisfactory accuracy consistent with the objectives and that it eventually boils down to the issue of building the right model. Malak R.J., and Paredis, C.J. identified three complementary validation problems, namely: (i) validity characterization, (ii) compatibility assessment, and (iii) adequacy assessment [13]. They claimed that individually these provide insight into the properties of predictive behavioral models, and together solve the validation problem and give sufficient information for validity descriptions.

Validation of objects, structures, and systems
Validation of physical objects typically involves empirical testing, whereas validation of (computer generated) virtual objects is based on computational analyses and simulations. Software validation extends to both the physical and the virtual domains. Many issues concerning system validation are in the focus of recent studies. To identify defects in the system specification, Brings, J. et al. dealt with the issue of supporting early validation of cyber-physical system (CPS) specifications based on modelbased prototype development [14]. Gonzalez, A.J. and Barr, V. exposed the differences to be taken into account in the verification and validation of intelligent systems [15]. Feth, P. et al. focused on the validation of open and heterogeneous systems, such as CPSs in the automotive domain, and proposed a simulation-based framework, which integrates AUTOSAR applications [16]. The virtual validation concerned the functional behavior and the performance of the software. Bradley, T.H. claimed that validation of system models within a multi-disciplinary design framework has three primary components (i) independent validation of the contributing analyses, (ii) validation of the overall performance and behavior on system-level, and (iii) validation for utility with respect to design decision making [17]. Goubault, E. et al. dealt with the precision problem of validation in the case of engineered systems and found that they can be precise enough only if the "semantics" of the physical environment is modeled accurately [18].

Validation of actions, processes, and services
Heikkinen, H.L. et al. proposed five principles for validation of actions in research: (i) principle of historical continuity, (ii) principle of reflexivity, (iii) principle of dialectics, (iv) principle of workability, and (v) principle of evocativeness [19]. White, J.D. offered criteria for validating interpretations of administrative actions and showed how action theory can be rescued from misplaced claims of subjectivism by locating the meaning of an act with the intention of the actor [20]. Validation of mental actions differs from validation of motor actions. For instance, in the case of inferring or reasoning, the prevailing reasoning model has a strong influence on the outcome of mental action [21]. Validation of a process checks if its plan and manifestation meet the predetermined specifications and quality attributes. The approach differs if it has done on the basis of the process itself or using a process-oriented model. Barth, A. et al. argued that validation is essential to design science and is best used by researchers to guide the development and evaluation of new methods [22]. As sketched up by Kelly, B. et al., the early service validation practice focused on the front end of service provisioning [23]. It checked if the service specification meets its requirements and interacts IOP Conf. Series: Materials Science and Engineering 573 (2019) 012108 IOP Publishing doi:10.1088/1757-899X/573/1/012108 5 sensibly with other services at the specification level. Combes, P., and Renard, B. reviewed the early approaches for validation of network services and exposed their dependence on the service specification, modeling, and implementation. Since service validation has rapidly gained importance, many specialized validation methods for digital and utility servicing have been proposed [24].

Validation of methods, methodologies, and tools
Method validation is one of the universally recognized elements of any rigorous research conduct [25]. For this reason, the number of techniques, protocols, and guidelines for research method validation is large. Their objective is to demonstrate whether the method is fit for a particular constructive or analytical purpose. Design methods are intangible artifacts. Consequently, their validation and testing require different approaches than those used in the case of tangible artifacts. In engineering contexts, methods have been proposed for both empirical and virtual validation. For instance, Mejí a-Gutié rrez, R. and Carvajal-Arango, R. discussed four types of prototypes that are used in design validation during product development, namely: (i) abstract prototypes, (ii) virtual prototypes, (iii) functional prototypes, and (iv) physical prototype [26]. Validation of computational methods is largely influenced by the purpose of their application and the purpose. There are both experimental and analytical (logical) approaches to the validation of computational methods. The approach should be logically rigorous, internally consistent, and mathematically correct. Olewnik, A.T., and Lewis, K.E. elaborated on validation of design decision methodologies [27]. According to them, the complexity of prescriptive models makes their validation a difficult task. Validation of a software engineering methodology was seen as a challenging issue by Lee, S.W., and Rine, D.C., since it involves testing of data, algorithms, usability, and performance [28]. Ahmad, W. et al. analyzed validation techniques for safety-critical software such as (i) functional failure analysis, (ii) HAZard and OPerability Studies (HAZOP), (iii) failure modes and effects analysis), and (iv) fault trees analysis, and found that HAZOP was the most prominent one for safety validation by the industrial stakeholders [29].

Concluding remarks
In the tradition of scientific inquiry, internal validation and external validation goes hand-in-hand. Because, a body of knowledge may suffer from internal biases due to lack of procedural rigor, and/or may not be appropriate for or has serious limitations with regards to its intended purpose. In the context of design methods, validation is driven by the question of whether the method provides design solutions 'correctly' (effectiveness) and whether it provides 'correct' design solutions (efficiency). Correct in this context are design solutions with acceptable operational performance, which are designed and realized with less cost and/or in less time. The above demonstrative survey provided evidence that validation works well as an alternative of mathematical modeling-based or real life experiments-based validation, which are difficult to realize in the case of design methods proposed by engineering research. The validity of a body of knowledge, that conveys the know-how of a design method, is determined by many more measures than just its usefulness. Validation has to do also with exploration and reduction of biases and errors (credibility provided by internal validation) and testing potentials and implications (transferability delivered by external validation). Nevertheless, usefulness is difficult to disprove as a pragmatic objective of validation of design methods. Furthermore, with regards to the validation of constructive methodologies, the literature offers only limited insights. A robust approach to rationally investigate the appropriateness of a design method for a specific purpose is needed. The rational robustness can be explained by a principle of logic, which hides in the logical background and provides a 'virtual' verification in an application independent fashion. This is the principle of syllogism that makes validation of design methods possible based on their outcome (i.e. reasoning with consequences). Consequently, a design method should be logically correct to result in logically correct results. If the outcome experienced as correct (i.e. not conflicting with the empirically observable reality), then the method was correct. Notwithstanding, it is also an issue that no biases are supposed to be created by the way of applying the method and its results.