{'fraction_non_alphanumeric': 0.033100135741881594, 'fraction_numerical': 0.007866068010163239, 'mean_word_length': 4.85769622833843, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Fundamental Considerations around Scenario-Based Testing for Automated Driving

The homologation of automated vehicles, being safety-critical complex systems, requires sound evidence for their safe operability. Traditionally, verification and validation activities are guided by a combination of ISO 26262 and ISO/PAS 21448, together with distance-based testing. Starting at SAE Level 3, such approaches become infeasible, resulting in the need for novel methods. Scenario-based testing is regarded as a possible enabler for verification and validation of automated vehicles. Its effectiveness, however, rests on the consistency and substantiality of the arguments used in each step of the process. In this work, we sketch a generic framework around scenario-based testing and analyze contemporary approaches to the individual steps. For each step, we describe its function, discuss proposed approaches and solutions, and identify the underlying arguments, principles and assumptions. As a result, we present a list of fundamental considerations for which evidences need to be gathered in order for scenario-based testing to support the homologation of automated vehicles.


I. INTRODUCTION
The introduction of automated vehicles (AVs) to public roads promises many benefits. These include a reduced number of accidents caused by driver errors (safety), increased efficiency of the transport system (environment), increased spare time for users (comfort) and mobility for elderly and impaired users (social inclusion) [1]. At higher levels of automation, in particular at SAE Level ≥ 3, AVs become complex autonomous systems operating in open context, i.e. dealing with unstructured real world environments. Thus, the validation and approval of AVs pose an enormous challenge [2]. As a newly emerging and safety-critical technology, rigorously proving safe operability and strictly avoiding accidents is crucial for societal acceptance.
The process of assuring functional safety for conventional road-vehicles is guided by the ISO 26262 [3]. This process has been extended by the ISO/PAS 21448 [4] to include the complementary aspect of the safety of the intended functionality (SOTIF), at least for SAE Level ≤ 2. In the automotive domain, testing is an important part of verification and validation and has traditionally used distancebased statistical arguments. However, for AVs, distancebased approaches to testing become infeasible due to the vast amount of distance that needs to be covered [5] [6]. Instead, other approaches begin to emerge, such as 'Responsibility Sensitive Safety' (RSS), which is a formal model that relies * The research leading to these results is partly funded by the German Federal Ministry for Economic Affairs and Energy within the project "VVM -Verification & Validation Methods for Automated Vehicles Level 4 and 5" and by the German Federal Ministry of Education and Research within the project "TESTOMAT -The Next Level of Test Automation" under grant agreement No. 01IS17026H. 1 OFFIS e.V., Oldenburg, Germany on defining a safety envelop around the vehicle [7]. However, it is questionable whether this formal model can actually be implemented in the real world [8]. A novel approach, which has been explored by recent research projects such as PEGASUS [9] and ENABLE-S3 [10], is to perform scenariobased testing (SBT). In SBT, testing is performed by deriving relevant test cases from a manageable set of scenario classes.
In this paper, we perform a deep dive into the fundamental considerations that must be taken into account in order for SBT to generate a meaningful contribution to the verification and validation of AVs. The goal is to identify underlying principles and assumptions which are essential to the successful realization of SBT for automated driving. We discuss relevant terminology, notation and a general framework around SBT in Section II. The introduced framework provides the structure for Section III. For each process step, we first describe the state of the art. Secondly, we semiformally examine the fundamental arguments, principles, and assumptions that are inherent to the aforementioned contemporary approaches. The core results of our analysis are summarized in Section IV as a condensed table, which may serve as an impulse for future research.

II. SCENARIO-BASED TESTING
Verification and validation serves the purpose of assuring properties of interest, such as safety and security, with respect to both the specified and intended purpose of the system. In the automotive domain, testing is defined as the 'process of planning, preparing, and operating or exercising an item or an element to verify that it satisfies specified requirements, to detect anomalies, and to create confidence in its behavior' [3]. In SBT, the central artifact for the operation is the scenario. Such testing approaches are pursued as state of the art in the AV testing community [11] [12].

A. Terminology and Notation
A standardized definition of the term scenario in the context of verification and validation of automated vehicles is presented in ISO/PAS 21448 [4], which we briefly introduce alongside a semi-formalism that will be used later on.
A scenario describes the development of scenes over time, thus building a temporal sequence. A test case is defined as a scenario enriched with pass/fail criteria suitable for test evaluation [4]. Menzel et al. propose the qualifications of functional, logical and concrete scenarios [13]. At the highest level of abstraction, a functional scenario is described semantically using natural language. Formalizing a functional scenario, a logical scenario L can be represented by a state space and its interrelations. It includes a list of relevant parameters A L = (α 1 , . . . , α n ) and their value ranges The state space of L can be described by V L = V 1 × · · · × V n ⊆ R n . Optionally, correlations between parameters and numeric constraints can be added. Probability distributions can be attached to the state space or the value ranges. We denote the set of all logical scenarios by L.
A concrete scenario C requires the assignment of a single value to each parameter. It can be obtained from a logical scenario L by instantiating all parameters α i ∈ A L with some v i ∈ V i , i.e. selecting a v ∈ V L . We denote the set of all concrete scenarios by C. Analogously, C L is the set of all concrete scenarios derived from a logical scenario L. The process of instantiating a set of logical scenarios to concrete scenarios can be seen as a map I : L → C, L → C ∈ C L w.r.t. the relations, constraints and distributions of L.

B. General Framework
In order to generate the aforementioned test cases and to interpret the test results, various steps are commonly executed up-and downstream of the actual testing process. Thus, it is crucial to embed SBT into a larger framework. Figure 1 shows such an abstract framework around SBT, mainly based on the ENABLE-S3 scenario-based verification and validation process [10] and the PEGASUS method [9]. Note that this framework is not meant to be a core contribution of our work, but rather an organizational tool for the following considerations obtained from incorporating various existing approaches. For example, the framework is also consistent with the publications by Huang et al.  Figure 7].
The first step, scenario elicitation, consists of deriving adequate scenario classes, e.g. in the form of logical scenarios. The requirement elicitation step complements these classes by a set of safety requirements that shall be satisfied by the system in the identified scenarios. Testing then evaluates the system's compliance to the requirements w.r.t. the scenario classes. Well-established concepts around testing can be found in standards such as the ISO/IEC/IEEE 29119 [16] and ISO 26262 [3]. Within this test process, test cases are initially derived from the scenario classes and safety requirements during test derivation. Subsequently, test execution is performed using an appropriate test bench, e.g. either virtually, physically or in a combination. The executed tests are assessed in a test evaluation step. The results of this whole process are then used in a subsequent, overarching safety argumentation, which contributes to the safety case.

III. FUNDAMENTAL CONSIDERATIONS
Structured along this framework, we examine arguments, principles and assumptions that, in the authors' opinion, are fundamental to the SBT process. We initially present some general considerations applicable to the overall process in Section III-A. Then, in Section III-B to Section III-F, we conduct a two-part examination for each process step: we first give an overview of the state of the art which is followed by a structured analysis of the underlying fundamental considerations. The key insights are marked with an ID, linking them to Table I. We omit an in-depth analysis of the safety argumentation, which will be subject to further research.

A. General Considerations
SBT requires a unified understanding of what the term scenario means. As mentioned in Section II-A, ongoing standardization activities work towards this unified understanding, but there remain open questions on the exact definition and usage of scenarios and what its qualifications describe. For example, the ISO/PAS 21448 uses logical scenarios in the test phase while Menzel et al. propose to use concrete scenarios [13]. Depending on these definitions, the technical realizations of these qualifications, e.g. OpenDRIVE 1 and OpenSCENARIO 2 , need to be sufficiently expressive and unambiguous such that scenario descriptions lead to test cases without loss or misinterpretation of information (G1).
Second, SBT often assumes smoothness of certain properties of interest, e.g. of criticality metrics, behaviors or trajectories, under variation of the parameters, i.e. slight variations of the parameters lead to slight changes of the considered property (G2). This smoothness can be exploited, e.g. to extrapolate knowledge of a single scenario to a set of scenarios or to explore the close proximity -in the sense of parameters -of a scenario in order to estimate the direction of more challenging scenarios. In both cases, this increases the certainty of the inference from finitely many test results to sets of scenarios when analyzing complex systems. However, Poddey et al. [2] argue that AVs are an instance of complex systems which rely heavily on sensor technologies for the perception of their open context. This poses difficulties, as processing the sensory input often uses machine learning. These algorithms are prone to misclassifications as a result of minor input changes [17], therefore smoothness of such classification algorithms cannot easily be assumed.
Since the open context in which AVs are supposed to operate is constantly evolving, the framework around SBT has to be adaptable. A complete set of scenario classes and testing methods at the time of elicitation may be incomplete at time of deployment due to a number of unforeseeable reasons, so called unknown unknowns [4] [18]. These range from novel objects or vehicles appearing in traffic over newly introduced traffic regulations up to massive structural changes in traffic due to human drivers reacting to the introduction of AVs. Hence, an integrated update process and constant monitoring of AVs during their life cycle are inevitable for SBT (G3).

B. Scenario Elicitation
Most published frameworks follow an expert-or datadriven approach to elicit functional or logical scenarios. Expert-driven approaches use a manifestation and formalization of expert knowledge about the real world. Proposed methods include the direct use of laws, regulations, system specifications, as well as already existing logical scenarios [19]. As an a priori formalization step, it has been proposed to incorporate such knowledge into ontologies [20]. Based on such formalized knowledge, experts can conduct safety analyses to identify hazardous scenarios [21]. Scenarios can also be elicited in a data-driven manner. For example, largescale observational studies can serve as a means to identify hazardous events, e.g. by learning a neural network [22]. Those can in turn be transformed into logical scenarios, e.g. by using the challenger framework [23], and can additionally be attached with probability distributions from the observed data [24]. Such a process can be enhanced by extending the already existing data basis [25]. Besides investigating large data sets, one can also closely investigate a concrete realworld drive [26], which then allows for a parametrization into a logical scenario. Finally, there exist attempts to combine expert-and data-driven frameworks. In top-down approaches, logical scenarios are classified by experts, e.g. on a keywordbased scheme [27], and the attached probability distributions are derived from real-world data. Orthogonally, bottomup approaches use expert analysis on data bases, e.g. a Successive Odds Ratio Analysis [28] or a classification tree approach [29], to classify observed concrete scenarios.
Expert-driven scenario elicitation necessitates defining a set of relevant real-world phenomena to be considered implicitly or explicitly, e.g. via an ontological representation. This approach is prone to creating a gap between identified and relevant phenomena. While this gap can be controlled by using systematic methods and made visible by explicit representations, completeness guarantees cannot be given. Moreover, experts lack the combinatorial power for the exhaustive exploration of the open context, increasing the risk of an incomplete set of scenario classes. In that regard, automated analyses are able to complement an expert-based approach (SE1). Most data-driven approaches rely on the existence of a sufficiently large data basis. It is assumed that relevant phenomena leave traces in recorded data, for example as causes of accidents in observational studies. In this case, relevant phenomena have to be present either directly in the data model or need to be manually identified by examination of the recorded scenarios (SE2). If the data basis is extended by collecting more data sets, the choice of observation is up to expert judgment and arguments must be made as to why this choice leads to scenario classes that enable SBT. Regarding the location of data collection, experts must argue that the chosen locations are in some form representative for the targeted operational design domain. This applies particularly to the validity of measured real-world distributions and their generalizability. For collecting data and evaluating concrete real-world drives, it has to be decided which entities and relations shall be observed in the real world. Additionally, appropriate tools and measurement sensors with sufficient accuracy have to be chosen purposefully and explicitly, in order for a downstream safety argumentation to reason over the validity of such decisions (SE3). Both expert-and data-driven approaches have in common that they either elicit an arguably complete classification of the scenario space, or need to reason on why the exclusion of certain scenarios is applicable in the current context (SE4). While a complete classification of scenarios that covers the entire scenario space is essential for SBT, such a decomposition does not reduce the overall test effort. However, if sound arguments can be made why some classes can be excluded altogether, omitting these classes significantly reduces the required test effort later on.

C. Requirement Elicitation
Of particular importance for any safety-relevant application is a systematic safety process that includes identification of hazards and risks, derivation of safety goals and decomposition of the high-level safety goals into safety requirements on item and component level [30] [31]. For systems up to SAE Level 2, this is well covered by ISO 26262 and ISO/PAS 21448. For SAE Level ≥ 3, scenarios appear to be an ideal starting point for the derivation and elicitation of acceptance criteria. Junietz et al. examine how such a quantified requirement definition can take place based on acceptable risk levels [32].
The process of identifying and specifying requirements is inherently bound to the problem of correctness (does the specification match the actual requirement), completeness (are all safety goals fully covered), consistency (the absence of contradictions) and validity (are we specifying a useful requirement in the first place) (RE1). Furthermore, for requirements tested virtually, specific arguments must be made with regards to measurability and computability. Here, the number of virtually executed scenarios can grow extremely large. Thus, the key elements of requirements need to be observable and measurable in the virtual environment (RE2).

1) Discretization:
During test derivation, a discretization step for a class L to a discrete classL is usually performed to reduce the test effort to a finite number of concretizations. This step can be seen as a map C L → CL with |CL| < ∞. If the state space V L of a logical scenario L, i.e. a whole scenario class, is fully described using a set of bounded value ranges V L = V 1 × · · · × V n ⊂ R n , some of these parameters describe measurable continuous realworld quantities. Examples contain the speed of an object and rainfall quantity. Mapping each continuous value range to a finite, discrete one, denoted by V i →Ṽ i , results in the aforementioned logical scenarioL with a finite, discrete state space VL =Ṽ 1 × · · · ×Ṽ n V L .
While overly fine discretizations may produce many redundant test cases [19], any discretization step leads to gaps in the original state space, possibly resulting in undiscovered safety violations [33]. One approach defines the map C L → CL according to probability distributions of the parameters [34], obtained from either synthesized or real-world data. In both cases, however, the question of the validity of the data arises. Discretization according to distributions is more sophisticated than equidistant splitting and leads to scenarios that resemble reality more closely, but its usefulness depends on the fundamental assumption that 'probability of occurrence of a scenario correlates with its relevance in terms of the safety argumentation' [34]. Whether this assumption is justified depends on the paradigm used for the safety argumentation [35]. We note that unlikely parameter combinations could lead to critical outcomes. Thus, discretization might lead to dismissing rare, critical scenarios due to preeliminated parameter combinations (TD1).
2) Variation Methods: In order to generate test cases for a given logical scenario L, it is necessary to instantiate the parameters with concrete values [13]. For efficient testing, a systematic approach for instantiation is required. This can be done either deterministically, e.g. by stepping (equidistantly) through V L or performing Boundary Value Analysis [36], or stochastically, e.g. using Monte Carlo methods [37]. If we attach probability distributions to each parameter value range V i (or a joint distribution defined on V L ), the instantiation I can be seen as a random variable that assigns concrete values v i ∈ V i to each parameter α i ∈ A L through sampling. Then, one may sample I according to the real-world distributions of the parameters w.r.t. all constraints and interrelations of L. It is likely that these distributions are approximated from either locally measured naturalistic data [38] or from valid simulated data [37] [34]. Both cases assume that sampling I according to real-world distributions enhances the value of test cases for the safety argument.
A different approach argues that scenarios being critical w.r.t. a suitable metric offer the most value for the safety argument and should therefore be tested preferably [39]. However, as these rarely occur naturally [32], one has to increase their significance during the sampling process. This can be realized by sampling I. Here, a criticality metric κ : S(C L ) → [0, 1], that maps a discrete time series of values T obtained from a simulation engine S to a scalar measuring the criticality of a concrete scenario C ∈ C L , can be optimized. This criticality-guided approach shifts much of the methodical burden to the metrics. First, a criticality metric κ has to reflect the real-world criticality accurately [39] (TD2). A single metric is unlikely to capture all types of critical phenomena so that several metrics in conjunction, each capturing different aspects of criticality, are required [40]. Moreover, each metric κ is evaluated on a time series T = S(C) depending on S, where C = I(L) is described by the parameters A L . Thus, sampling I in order to optimize κ can only capture real-world criticalities that are (i) actively influenced by the parameters A L and (ii) accurately computed by S. Clearly, (i) depends on the scenario description language and (ii) hinges on the validity of the simulation environment and the utilized models. Influencing factors for real-world criticality whose effects are not conveyed by this process can thus not be revealed. Simple metrics such as TTC incorporate only few parameters, but can be optimized efficiently. More involved metrics cover more influencing factors at the cost of higher computational effort [41]. When defining a new metric, e.g. by combining known metrics and taking a weighted sum, the complexity of the associated optimization problem is determined by its mathematical properties, e.g. being continuous, differentiable, or convex.
For each criticality metric κ a threshold c κ ∈ [0, 1] is required in order to label a time series T = S(C) as critical whenever κ(T ) ≥ c κ (TD3). Junietz et al. [42] suggest fitting thresholds based on manually annotated scenarios which are then used to learn binary classifiers. Criticality thresholds can be used to define the critical subspace C L,κ,S = {C ∈ C L | κ(S(C)) ≥ c κ } ⊂ C L w.r.t. κ and S. The question arises whether optimization algorithms reliably cover the set C L,κ,S by finding critical concrete scenarios C ∈ C L,κ,S from all of its connected components. If components of C L,κ,S are missed entirely, one can no longer argue about testing all critical instantiations of L (TD4). In order to prevent this, optimization algorithms may have to be executed repeatedly, using varying step lengths or starting points. In any case, the dependency on the simulation engine S remains.

E. Test Execution
Depending on the stage of the development process, there exists a diverse set of test execution methods [14]. An essential strategy is mixed virtual-physical testing, i.e. replacing physical components with virtual ones [43]. For each involved component and simulation model, a test method needs to be chosen that provides valid test results [44].
In order for virtual testing to replace physical testing, the corresponding virtual models have to be verified and validated against their physical counterparts [45]. The validity of a simulation environment depends on the validity of many

ID Consideration Process
Step Literature

G1
The term scenario, its qualifications and technical realizations are sufficiently expressive and unambiguous.

G2
If smoothness of a property of interest of the system is assumed, this assumption is asserted.

G3
The SBT process is adaptable to real-world changes.
SE1 An expert-based approach is executed systematically and supported by automation. SE2 An identification of all relevant phenomena is facilitated.
SE3 A data-driven approach uses representative measurement locations, devices and valid probability distributions.
SE4 A decomposition of the test space into scenario classes is complemented either by evidence for its completeness or an argumentation for the omission of classes. individual models (e.g., environment models, behavior models, sensor models) as well as their interactions. Gathering relevant real-world data for virtual models is a key challenge in such a validation process, which also depends on the employed notion of validity (cf. [46]). Thus, the question how to validate simulation environments remains subject to current research (TX1).

F. Test Evaluation
After executing test cases, they need to be evaluated according to safety requirements. For SBT, these requirements can appear in the form of thresholds for criticality metrics, being evaluated using a finite sequence of measurements during the evolution of a concrete scenario [47]. It is possible to judge compliance of real-world tests with qualitative requirements using experts. For virtual batchtesting of scenarios, however, qualitative requirements need to be mapped to criticality metrics and thresholds. PEGASUS suggested a 4-stage-process for test case evaluation that evaluates the aspects safety distances, absence of collision, causality and mitigation [48]. Instead of binary test results, [35] proposes test case evaluation on a more detailed ordinal scale, incorporating knowledge about possible discrepancies between the anticipated and actual category of a test case depending on the underlying paradigm.
Formalizing test evaluation, we assume that executing a concrete scenario C instantiated by I from L results in a discrete time series of values T . This time series can either be obtained by a simulation engine S or by realworld measurements. We focus on the former here, i.e. C = I(L) and T = S(C). A pass/fail-criterion κ(·) < c κ is denoted as R w.r.t. a suitable criticality metric κ and threshold c κ for L. The results of testing N instantiations C 1 , . . . , C N of L, obtained from R, need to be aggregated. A key challenge is to obtain a useful aggregate statement to support the safety case (TE1). While a simple success ratio 1 N N i=1 1 {S(Ci)|=R} can be adequate for comfort functions, it might not be a good choice for testing safety goals. This challenge becomes even more complex when evaluating multiple metrics.
The map I and also S -for a probabilistic simulation -can be seen as stochastic processes. Their composition with a criticality metric κ is a random variable X = κ • S • I : L → [0, 1]. For test evaluation of an entire scenario class L, for which not all instances can be tested, we need to make a statistical statement about the system under test failing a test coming from L (TE2). A possible strategy is to estimate the probability of a random scenario derived from L being critical w.r.t. κ, i.e. p = p(κ, L, S, I) = P (X ≥ c κ ). The true distribution of p is unknown which requires an estimatep. Assuming the tested system is well-engineered and the threshold c κ is well-chosen, we can expect p to be very small. As explained in Section III-D, advanced Monte Carlo methods can be applied to obtain an estimatep. A confidence statement about p usingp can then be interpreted as a statistical statement about test coverage based on the generated samples [49].

IV. RESULTS
As the result of our analysis, we present Table I, which summarizes the fundamental considerations from Section III, marked with an identifier. These considerations are grouped according to the steps of the SBT process, as depicted by Figure 1, and each step is annotated with examples of the relevant literature.
In order to obtain a valid safety case for the release of an AV, a coherent well-structured safety argumentation is required. Such a safety case needs to argue why and how the provided evidence satisfies the high-level safety requirements. One integral part of this evidence are the results of the testing process. In this regard, Table I provides a collection of considerations for which meaningful evidences need to be gathered in order for SBT to support a safety case. However, this collection is not exhaustive and additional considerations may be necessary to establish a valid safety case.

V. CONCLUSION
A comprehensive review and analysis of the literature concerning SBT for automated vehicles was performed. We presented numerous arguments, principles and assumptions that are fundamental to the automotive SBT approach. For each step of the process, we analyzed the strengths and weaknesses of the most promising contemporary approaches in order to uncover potential gaps and inconsistencies. As a result, we obtained a collection of fundamental considerations that need to be substantiated with evidences. It is subject to further research to provide approaches that generate these evidences reliably. Finally, an exploration of additional fundamental considerations complementing our collection is likely to be necessary.